光谱工具集:采集、回放、寻峰、波长标定
- spectro_core.py: 共享核心(加载/标定/基线/寻峰/高斯拟合/双峰联合拟合/CSV-JSON) - peak_finder.py: 寻峰与峰信息导出,--pick 交互选峰(含 RAW 预览) - wavelength_calibration.py: 改用 core 寻峰,汞灯 577/579 双峰可分 - wavelength_player.py / hq_average.py: 复用 core 的标定与解码加载
This commit is contained in:
commit
45249b59f9
24 files changed
+11449
No files matched your search
@@ -0,0 +1,14 @@
|
||||
# 行尾统一:仓库内文本一律 LF(Windows 检出仍可用)
|
||||
* text=auto
|
||||
*.py text eol=lf
|
||||
*.md text eol=lf
|
||||
*.txt text eol=lf
|
||||
*.csv text eol=lf
|
||||
*.json text eol=lf
|
||||
*.ps1 text eol=crlf
|
||||
|
||||
# 二进制,不做行尾转换
|
||||
*.png binary
|
||||
*.jpg binary
|
||||
*.npy binary
|
||||
*.bin binary
|
||||
+33
@@ -0,0 +1,33 @@
|
||||
# ---- 采集数据:体积大,不入库 -------------------------------------------
|
||||
recordings/
|
||||
*.bin
|
||||
*.jsonl
|
||||
|
||||
# ---- 运行产物 -----------------------------------------------------------
|
||||
peak_exports/
|
||||
*.npy
|
||||
*.log
|
||||
|
||||
# ---- Python -------------------------------------------------------------
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*.so
|
||||
.venv/
|
||||
venv/
|
||||
env/
|
||||
.pytest_cache/
|
||||
.mypy_cache/
|
||||
.ruff_cache/
|
||||
|
||||
# ---- 编辑器 / 系统 ------------------------------------------------------
|
||||
.vscode/
|
||||
.idea/
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
desktop.ini
|
||||
|
||||
# ---- 临时文件 -----------------------------------------------------------
|
||||
*.tmp
|
||||
*.bak
|
||||
*.orig
|
||||
*.rej
|
||||
+283
@@ -0,0 +1,283 @@
|
||||
# 寻峰与峰信息导出(peak_finder.py)
|
||||
|
||||
`peak_finder.py` 在 `hq_record.py` 录制的 RAW 数据里找谱线峰:用 `hq_spectrum.py`
|
||||
的解码和降维方式得到 R/G/B 三条谱,扣除局部基线后寻峰、拟合、给出峰形和**像素坐标**,
|
||||
最后把**选中的峰**导出为图片、CSV、JSON 和原始像素块。
|
||||
|
||||
**本程序不做标定。** 它测的是传感器列坐标(`sensor_x_px`);只有当你在命令行里显式给出
|
||||
`--calibration <某个 calibration.json>` 时,才会按该文件里的系数把列坐标换算成波长
|
||||
(`wavelength_nm = polyval(coefficients_descending, sensor_x_px)`)。不给就只输出像素,
|
||||
波长列留空。这会避免把某一次实验的标定悄悄套到别的录制上——标定是否适用由你判断,
|
||||
依据(系数、确认范围、参考录制)会原样写进 `peaks.json`。
|
||||
|
||||
支持的数据是 `hq_record.py` 录制的 RAW(`BGGR_PISP_COMP1`、2028×1520、uint8 容器)。
|
||||
逐帧元数据里的黑电平(`SensorBlackLevels`,本数据集为 4096)会被自动使用。
|
||||
|
||||
依赖:Python 3 + NumPy + Matplotlib,**不需要 SciPy**。脚本放在 `hq_spectrum.py`
|
||||
同目录即可(本目录里那份 `hq_spectrum.py` 是 2026-11-14 版的副本,方便单独运行;
|
||||
你自己的 `hq_spectrum.py` 更新后记得同步过来,或用 `--module-dir` 指过去)。
|
||||
|
||||
兼容性:与 2026-10-24 版和 2026-11-14 版 `hq_spectrum.py` 都测过。新版把 PC1B 解码器
|
||||
改成了按需处理 qmode 3 + 直接写入 uint16(实测三份录制逐像素完全一致),并重写了
|
||||
`make_preview`——后者现在要求**整数** Bayer 帧,本程序已改为把原始 uint16 帧(多帧
|
||||
平均时是四舍五入后的均值)交给它。
|
||||
|
||||
## 快速开始
|
||||
|
||||
在 `peak_finder.py` 所在目录运行。先设两个路径变量(换成你自己的),下面直接引用:
|
||||
|
||||
```powershell
|
||||
$rec = "C:\Users\peng0\Dev\Light\recordings\hq_20261101_114417" # 录制目录
|
||||
$cal = "C:\Users\peng0\Dev\Light\Programs\wavelength_calibration\calibration.json"
|
||||
|
||||
# 1. 先只看找到了哪些峰(不导出图片,速度快)
|
||||
python peak_finder.py --input $rec --select none
|
||||
|
||||
# 2. 导出全部峰(每峰一张峰形图 + CSV + 原始像素块)
|
||||
python peak_finder.py --input $rec --select all
|
||||
|
||||
# 3. 只导出 540–560 nm 的峰和最强的两条(按波长选就必须给 --calibration)
|
||||
python peak_finder.py --input "C:\Users\peng0\Dev\Light\recordings\hq_20261110_230441" `
|
||||
--calibration $cal --select nm:540-560,top:2
|
||||
|
||||
# 4. 按编号导出(编号以上一次输出的 P001… 为准)
|
||||
python peak_finder.py --input $rec --select id:2,id:4-5
|
||||
|
||||
# 5. 在窗口里点选:左边全谱,右边跟着鼠标显示该峰的 RAW 像素块和放大谱形
|
||||
# 悬停预览、点击切换选/不选,a=全选,n=全不选,e=导出,q=取消
|
||||
python peak_finder.py --input $rec --pick
|
||||
|
||||
# 6. 平均多帧降噪;只取 ROI 中的一段行,减小谱线倾斜造成的展宽
|
||||
python peak_finder.py --input "C:\Users\peng0\Dev\Light\recordings\hq_20261111_092116" `
|
||||
--frame 5 --average 8 --rows 600,760
|
||||
|
||||
# 7. 顺带对照汞灯谱线表(需要波长,所以也给标定)
|
||||
python peak_finder.py --input $rec --calibration $cal --match line_tables\mercury_hg1.csv
|
||||
```
|
||||
|
||||
只有第 3 条和第 7 条给了 `--calibration`;其余只输出像素坐标,要波长自己补上。
|
||||
给了 `--calibration` 时终端会打印:
|
||||
|
||||
```
|
||||
id nm x_px ch SNR height FWHM px FWHM nm slit y flags
|
||||
* P001 435.6458 302.489 RB 517 2983 18.08 5.151 805.571 extrapolated, Hg I -0.19 nm
|
||||
* P002 545.8237 689.157 RGB 2925 13545 10.35 2.949 763.262 saturated, Hg I -0.25 nm
|
||||
P003 579.5502 807.519 RGB 939 4346 14.74 4.2 771.807 Hg I +0.48 nm
|
||||
```
|
||||
|
||||
(这三条相对 NIST 汞灯 435.8328 / 546.0735 / 579.0663 nm 的偏差为 −0.19 / −0.25 / +0.48 nm,
|
||||
属于该标定本身约 0.7 nm RMS 的水平。)
|
||||
|
||||
## 交互选峰窗口(`--pick`)
|
||||
|
||||
```
|
||||
┌───────────────────────────────┬──────────────────────┐
|
||||
│ │ RAW 预览(整幅、与 │
|
||||
│ 全谱(选中/未选中/聚焦) │ 播放器同款)+ROI 框 │
|
||||
│ │ +峰值所在列 │
|
||||
│ ├──────────────────────┤
|
||||
│ │ 放大谱形 + 拟合曲线 │
|
||||
│ │ + 半高宽标记 │
|
||||
└───────────────────────────────┴──────────────────────┘
|
||||
```
|
||||
|
||||
鼠标移到某条谱线上,右边两块立刻切换成那个峰。右上就是 `hq_spectrum` 播放器里那张
|
||||
auto-stretched RAW 预览(缩放到 ROI 区域):橙色框是 ROI,青色竖线是当前峰的传感器列,
|
||||
青圈是它在缝方向的行位置;右下是放大的净信号和拟合曲线。
|
||||
|
||||
点一下切换选中(红色为选中、灰色为未选中、线更粗的是当前聚焦)。`a` 全选、`n` 全不选、
|
||||
`e` 导出并关闭、`q` 取消。见 `pick_window.png`。
|
||||
|
||||
窗口用固定的三块布局(不依赖 constrained_layout),可以随便拉伸;如果窗口比屏幕还大,
|
||||
程序会先按屏幕尺寸缩一档,避免开出来就有一半在屏幕外。
|
||||
|
||||
> 导出的单峰 PNG 里那小块仍然是**未去马赛克的 Bayer 像素块**(`inferno` 色阶),
|
||||
> 用来看像素级的线形和倾斜;想看"谱线在相机画面里的位置",看这个选峰窗口或
|
||||
> `peaks_overview.png`。
|
||||
|
||||
## 参数
|
||||
|
||||
### 输入
|
||||
|
||||
| 参数 | 默认 | 说明 |
|
||||
| --- | --- | --- |
|
||||
| `--input` | 必填 | 录制目录(含 `meta.json`、`frames.bin`) |
|
||||
| `--frame` | `0` | 帧号(0 起) |
|
||||
| `--average` | `1` | 从 `--frame` 起平均多少帧;>1 时导出的像素块是均值(float32) |
|
||||
| `--roi` | `auto` | `auto` 自动检测光谱带;`x0,y0,x1,y1` 手动;`calibration` 用标定 JSON 里保存的 ROI |
|
||||
| `--rows` | 无 | 在 ROI 内再限制行范围 `y0,y1`(全传感器行号),可减小谱线倾斜展宽 |
|
||||
| `--reduce` | `mean` | 沿缝方向的行合并方式 `mean`/`median` |
|
||||
| `--black` / `--full-scale` | 自动 | 覆盖黑电平/满量程(解码后 ADU) |
|
||||
| `--calibration` | 无 | 要换算波长时显式给出的标定 JSON;不给则只输出像素坐标(`wavelength_nm` 留空) |
|
||||
| `--wavelength-offset` | `0` | 给所给标定加一个固定波长偏移(nm),仅用于已确认的系统偏移;本身就是标定假设,慎用 |
|
||||
| `--module-dir` | 脚本目录 | `hq_spectrum.py` 所在目录 |
|
||||
| `--quiet` | 关 | 不打印过程信息 |
|
||||
|
||||
### 寻峰
|
||||
|
||||
| 参数 | 默认 | 说明 |
|
||||
| --- | --- | --- |
|
||||
| `--baseline-window` | `100` | 形态学开运算基线的窗口(传感器像素);`0` 表示不减基线 |
|
||||
| `--peak-distance` | `4` | 同通道两峰最小间距(传感器像素,2 px = 一个采样点),再近就只留高的;调大用于合并"勉强分开"的混合峰 |
|
||||
| `--peak-height` | `0.02` | 最小峰高,占该通道最大净信号的比值 |
|
||||
| `--peak-prominence` | `0.01` | 最小地形突出度,占该通道最大净信号的比值 |
|
||||
| `--min-snr` | `15` | 最小峰高,单位是该通道噪声估计 |
|
||||
| `--max-peaks` | `40` | 每通道最多峰数 |
|
||||
| `--group-tol` | `4` | 跨 R/G/B 合并成同一条谱线的距离(像素) |
|
||||
| `--fwhm-span` | `60` | 半高宽搜索的最大半程(像素);超出记为无半高宽 |
|
||||
| `--fit-window` | `0` | 拟合窗半宽(像素);0 自动按半高宽定 |
|
||||
| `--shape` | `gauss` | 线型模型 `gauss`/`lorentz` |
|
||||
|
||||
### 选择与导出
|
||||
|
||||
| 参数 | 默认 | 说明 |
|
||||
| --- | --- | --- |
|
||||
| `--select` | `all` | `all`、`none`、`3`(编号)、`3-7`(编号段)、`nm:540-560`(波长段)、`546.1`(波长±典型线距的一半)、`top:5`(SNR 前 5),可用逗号组合取并集 |
|
||||
| `--pick` | 关 | 开窗口点选:左边全谱,右边实时显示鼠标所指峰的 RAW 像素块与放大谱形;悬停预览、点击切换选中 |
|
||||
| `--output` | `peak_exports/<录制名>_f<帧号>` | 输出目录;单帧是 `_f0003`,`--average 8` 时是 `_f0003-0010_avg8` |
|
||||
| `--patch-half-width` | `24` | 导出原始像素块的横向半宽(像素) |
|
||||
| `--patch-format` | `npy` | 像素块格式 `npy`/`csv`/`both`/`none` |
|
||||
| `--patch-csv` | 关 | 在 npy 之外再写一份长表 CSV |
|
||||
| `--no-images` | 关 | 不生成每峰的 PNG |
|
||||
| `--match` | 无 | 谱线表(CSV/JSON),用于给出谱线身份建议 |
|
||||
| `--match-tolerance` | `1.5` | 匹配容差(nm) |
|
||||
|
||||
## 导出内容
|
||||
|
||||
```
|
||||
peak_exports/hq_20261101_114417_f0000/
|
||||
├─ peaks_overview.png 全谱:上图原始信号+基线,下图归一化净信号+峰标注
|
||||
├─ peaks_all.csv 所有通道的全部候选峰(每行一个通道峰)
|
||||
├─ selected_peaks.csv 选中的峰,一行一条物理谱线(只包含像素坐标等汇总)
|
||||
├─ peaks.json 清单:录制、帧、ROI、黑电平、噪声、标定、阈值、每峰的逐通道细节
|
||||
└─ peaks/
|
||||
├─ peak_P002.png 峰形图:全谱定位 + RAW 像素块 + 放大曲线与拟合
|
||||
├─ peak_P002_shape.csv 峰形数据:逐通道 raw / 基线 / net / 拟合值 / 残差
|
||||
└─ peak_P002_raw_y572-978_x664-714.npy 该峰的 RAW 像素块(uint16;文件名即像素框)
|
||||
```
|
||||
|
||||
`peak_Pxxx.png` 三块内容:上方是该峰在全谱中的位置;左下是把该峰附近的 RAW Bayer
|
||||
像素块(未去马赛克,横轴/纵轴都是真实传感器行列号);右下是放大后的净信号、拟合曲线
|
||||
(虚线)、半高宽标记,图例里的 `x±…` 是拟合中心的误差。标题给出波长、传感器列、通道、
|
||||
SNR、FWHM;饱和峰带 `[SATURATED]`,标定范围外带 `[extrapolated]`,重叠双峰经联合拟合
|
||||
分开的带 `[pair fit]`。
|
||||
|
||||
### selected_peaks.csv 主要字段
|
||||
|
||||
| 字段 | 含义 |
|
||||
| --- | --- |
|
||||
| `id` | 峰编号 `P001`…(按传感器列排序,同一数据同参数可复现) |
|
||||
| `wavelength_nm` | 波长(空气,nm);无标定时为空 |
|
||||
| `extrapolated` | 1 = 在标定确认范围之外,属于外推 |
|
||||
| `channels` / `primary_channel` | 该线出现在哪些 CFA 通道 / 用哪个通道出指标 |
|
||||
| `sensor_x_px` | 峰中心的全传感器列坐标(亚像素,加权合并各通道拟合中心) |
|
||||
| `center_spread_px` | 各通道中心的最大差,判断跨通道一致性 |
|
||||
| `sample_index` / `sample_step_px` | 该峰在降维谱数组里的下标 / 采样步长(2 px) |
|
||||
| `dispersion_nm_per_px` | 该处色散 |
|
||||
| `height_net_adu`、`snr`、`fwhm_px`、`fwhm_nm`、`area_net_adu_px` | 峰高(扣基线)、信噪比、半高宽(像素与 nm)、净面积 |
|
||||
| `baseline_adu`、`raw_adu`、`saturated` | 基线值、峰顶原始值、是否饱和 |
|
||||
| `slit_y_px` | 该谱线在缝方向(行方向)的强度加权中心行 |
|
||||
| `roi_x0..roi_y1` | 使用的 ROI(全传感器坐标,半开区间) |
|
||||
| `patch_x0..patch_y1` | 导出的 RAW 像素块框(全传感器坐标,半开区间) |
|
||||
| `frame`、`frames_averaged` | 帧号、平均帧数 |
|
||||
| `image_png`、`shape_csv`、`raw_npy`、`raw_csv` | 对应文件(相对输出目录) |
|
||||
| `match_species`、`match_nm`、`match_delta_nm` | 谱线表匹配结果(未启用时为空) |
|
||||
|
||||
`peaks_all.csv` 另外给出逐通道的 `center_parabola_px`、`center_fit_px`、
|
||||
`center_err_px`(拟合中心的形式误差)、`fwhm_halfmax_px`(半高宽直接测量值,可与拟合值
|
||||
对照)、`fit_ok`、`fit_rms_adu`、`deblended`,便于判断某个峰的拟合是否可信。实际采用的
|
||||
中心由 `center_source` 标明:`fit` = 高斯/洛伦兹拟合中心,`apex` = 拟合被拒后用峰顶三点
|
||||
抛物线,`sample` = 平顶/饱和导致抛物线也不可用时的采样点。重叠双峰的两个分量共用同一个
|
||||
`fwhm_px`(见下),因此两者线宽相同是正常的。
|
||||
|
||||
## 坐标约定
|
||||
|
||||
* 全部为 **0 起** 的全分辨率传感器坐标;像素 `x` 覆盖 `[x, x+1)`,第 0 行在画面顶部。
|
||||
* 降维谱每 2 px 一个采样点(CFA),拟合中心可以落在采样点之间,因此 `sensor_x_px` 是小数。
|
||||
* `roi`、`patch` 都是半开区间 `[x0,x1) × [y0,y1)`,与 `hq_spectrum.py` 的 ROI 定义一致。
|
||||
* `peak_*_raw_y572-978_x664-714.npy` 的第 0 行 = 传感器第 572 行,第 0 列 = 第 664 列
|
||||
(文件名里就是行列范围),可以直接 `np.load` 后与 `patch_x0…patch_y1` 对上。
|
||||
|
||||
## 方法说明
|
||||
|
||||
1. 用 `hq_spectrum.py` 的 `RecordingSource` 解码一帧(或若干帧均值):PC1B 解压成
|
||||
uint16 Bayer 图。
|
||||
2. `extract_planes` 按 BGGR 拆成 B、Gb+Gr(平均)、R 三个平面,沿缝方向(行)取
|
||||
mean/median,得到一个像素一个采样点的三条谱;x 用 `channel_x_positions` 换算成
|
||||
全传感器列号(G 在 `x0+0.5+2k`,R/B 在 `x0+dx+2k`)。
|
||||
3. 基线:对每条谱做形态学开运算(先最小值滤波再最大值滤波,窗口默认 ±50 采样点),
|
||||
再去掉开运算结果的高频毛刺。它只用于**测量峰高/峰宽**,原始值单独导出。
|
||||
4. 噪声:中位绝对一阶差分 / (0.6745·√2),对孤立尖峰不敏感。
|
||||
5. 候选峰:局部极大值(平台取中点),要求峰高 ≥ max(高度比×通道最大, SNR 阈×噪声),
|
||||
且**地形突出度**(topographic prominence,向上走到比它更高的点为止,取两侧谷底较高者)
|
||||
达标,并与更强的峰间距 ≥ `--peak-distance`。用真实突出度而不是固定窗口,肩峰会被
|
||||
自动排除,而真正分得开的双峰不会被并掉。
|
||||
6. 精化:加线性背景的高斯/洛伦兹拟合(自实现的 Levenberg–Marquardt,带边界投影),
|
||||
同时给出参数的形式误差。拟合不合格(不收敛、宽度顶到边界、残差 RMS > 20% 峰高)时
|
||||
**退回峰顶**(三点抛物线,`center_source=apex`),平顶/饱和导致抛物线也不可用时再用
|
||||
检测到的采样点。**全程不使用质心**:质心是窗口的函数而不是峰的函数,同一个峰换个窗口
|
||||
就换个值(本数据集里实测偏过 6 px 和 102 px),不能当峰位置用。
|
||||
|
||||
7. **双峰去混叠**:若两条相邻谱线的间距小于约 1.2 倍平均半高宽,就用「两个峰 + 一个
|
||||
公共线宽 + 线性背景」做联合拟合。单峰拟合在重叠区会把两个中心都往中间拉(实测汞
|
||||
577/579 会合成一条 579.5 nm),联合拟合才能把它们分开;标 `deblended=1`、`[pair fit]`。
|
||||
8. 跨通道合并:按**检测到的极大值位置**聚类(容差 `--group-tol`),同一通道只留 SNR
|
||||
最高者;同一条线在不同通道被检测到位置差几像素时,按拟合中心再合并一次
|
||||
(容差 1.5 px);峰中心按净峰高加权平均。
|
||||
9. 波长(**仅当你给了 `--calibration`**):`wavelength_nm = np.polyval(coefficients_descending,
|
||||
sensor_x_px)`,与 `wavelength_player.py` 用的是同一个公式;超出该标定确认范围的峰标
|
||||
`extrapolated`。不给标定时这一列与 nm 相关的量都留空,图上的 x 轴就是传感器列。
|
||||
|
||||
## 已知限制
|
||||
|
||||
* **标定只对同一光学结构和传感器模式有效。** 标定来自 `hq_20261101_114417`,其确认范围
|
||||
是 436–580 nm;其它录制的同一根谱线位置可能相差几个像素(几百分之一 nm 到 1 nm)。
|
||||
需要更准就重新跑 `wavelength_calibration.py`。`--wavelength-offset` 只能在已确认是
|
||||
固定系统偏移时使用,不能代替重新标定。
|
||||
* **谱线沿行方向不是一条直线(弯曲 + 漂移)。** 实测汞 546.1 nm(`hq_20261101_114417`):
|
||||
照亮带(约 363 行)上谱线中心是一条 S 形曲线,峰峰值 9.6 px(≈2.7 nm),按强度加权的
|
||||
1σ 是 1.76 px,而**净线性漂移只有 2.6 px(0.41°)**。因此沿全部 970 行平均得到
|
||||
FWHM 10.35 px(2.95 nm),只用最亮的 70 行(`--rows 690,760`)得到 7.37 px(2.10 nm),
|
||||
中心相差 1.76 px(0.50 nm)。要测真实线宽就用 `--rows` 取窄行窗口。
|
||||
导出的 RAW 像素块只有 ~50 px 宽却横跨几百行,`imshow` 的 `aspect="auto"` 会把横向
|
||||
放大约 10 倍,看着比实际"斜"得多——上面的数字才是真实几何。
|
||||
* **饱和峰**:核心像素达到 65535 时标 `[SATURATED]`,此时峰高、线宽和中心都可能偏。
|
||||
降低曝光/增益或缩短曝光时间重新录制更可靠。
|
||||
* **重叠双峰要小心。** 汞 576.96/579.07 nm 这类双线在默认 ROI 下间距只有约 6 px,
|
||||
而单条线 FWHM 约 7.5 px。程序会用公共线宽做联合拟合把它们分成两条(本例
|
||||
P004≈578.0 nm / P005≈580.4 nm),但分量位置的不确定度比孤立线大一个量级:
|
||||
`center_err_px` 约 0.2–0.5 px,而孤立线只要 0.02–0.3 px。间距小于约 1 倍半高宽时
|
||||
结果对线型假设很敏感,把它当"参考位置"而不是精密测量值。
|
||||
* **弱线/连续谱**:`--peak-height`、`--peak-prominence`、`--min-snr` 是相对的,暗弱录制
|
||||
里需要适当调低;连成一片的宽峰会只报一个中心,或被 `--peak-distance` 合并。想把
|
||||
靠得很近的线分开就把 `--peak-distance` 调小(最小 2 px = 一个采样点)。
|
||||
* 基线是形态学开运算的结果,宽于基线窗口的发射带会被部分当作基线;需要时可加大
|
||||
`--baseline-window` 或设为 0。
|
||||
* 波长单位是空气波长,未做光谱响应(强度)校正,强度仍是相机 ADU。
|
||||
|
||||
## 与其它脚本的关系
|
||||
|
||||
| 脚本 | 作用 |
|
||||
| --- | --- |
|
||||
| `hq_spectrum.py` | 解码/实时显示/回放的核心,本脚本复用它的解码、ROI、CFA 分离 |
|
||||
| `hq_record.py` | 录制 RAW |
|
||||
| `wavelength_calibration.py` | 用汞灯重新做像素→波长标定,产出 `wavelength_calibration/`;peak_finder 只读它的 JSON,从不改它 |
|
||||
| `wavelength_player.py` | 带标定的回放 |
|
||||
| `hq_average.py` | 多帧平均成新录制(本脚本用 `--average` 做同样的事,只是不落盘) |
|
||||
| `peak_finder.py` | 本脚本:寻峰 + 峰形/像素坐标导出 |
|
||||
|
||||
本脚本**只读取**标定,不会修改 `wavelength_calibration/` 里的任何文件。
|
||||
|
||||
## 测试
|
||||
|
||||
```powershell
|
||||
# 录制不在程序目录附近时,用环境变量指过去,否则端到端用例会自动跳过
|
||||
$env:PEAK_FINDER_TEST_RECORDING = "C:\Users\peng0\Dev\Light\recordings\hq_20261101_114417"
|
||||
$env:PEAK_FINDER_TEST_CALIBRATION = "C:\Users\peng0\Dev\Light\Programs\wavelength_calibration\calibration.json"
|
||||
python -m unittest test_peak_finder -v
|
||||
```
|
||||
|
||||
覆盖基线/噪声估计、亚像素中心、高斯/洛伦兹拟合、候选筛选、跨通道合并、选择语法、
|
||||
标定校验、谱线表匹配,以及一次真实的端到端导出(录制不在时会自动跳过)。
|
||||
|
||||
|
||||
@@ -0,0 +1,86 @@
|
||||
# spectrometer
|
||||
|
||||
Raspberry Pi HQ Camera(IMX477)RAW 光谱工具集:录制、回放、寻峰、波长标定。
|
||||
|
||||
采集的是相机 **RAW** 数据(`BGGR_PISP_COMP1`,2028×1520,16 bit 容器),不做 ISP、
|
||||
不去马赛克、不做平滑;所有分析都基于按 CFA 分离后的 R/G/B 三条一维谱。
|
||||
|
||||
## 文件
|
||||
|
||||
| 文件 | 作用 |
|
||||
| --- | --- |
|
||||
| `hq_spectrum.py` | 核心库 + 实时播放器:PC1B 解码、CFA 分离、ROI、降维、预览、交互窗口 |
|
||||
| `hq_record.py` | 把相机 RAW 流录成 `recordings/<时间戳>/{frames.bin,frames.jsonl,meta.json}` |
|
||||
| `hq_average.py` | 多帧平均成一条新录制(降噪) |
|
||||
| `spectro_core.py` | **共享核心**:模块加载、标定解析、基线/噪声/寻峰/拟合/双峰分解、CSV/JSON 输出 |
|
||||
| `peak_finder.py` | 寻峰与峰信息导出:峰形图、峰形数据、像素坐标、RAW 像素块;`--pick` 交互选峰 |
|
||||
| `wavelength_calibration.py` | 用汞灯谱线标定 像素→波长,产出 `wavelength_calibration/` |
|
||||
| `wavelength_player.py` | 带已存标定的回放(波长轴、瀑布图) |
|
||||
| `PEAK_FINDER.md` / `WAVELENGTH_PLAYER.md` | 两个工具的详细说明 |
|
||||
| `test_peak_finder.py` / `test_wavelength_player.py` | 单元测试(含真实录制的端到端用例) |
|
||||
| `wavelength_calibration/` | 一次汞灯标定的结果:`calibration.json` 及残差、候选、逐像素波长表 |
|
||||
|
||||
`anal.py` 是 2026-10-23 的 `hq_spectrum.py` 旧副本,函数一一对应、无独有内容,属于
|
||||
遗留文件,可删。
|
||||
|
||||
## 依赖
|
||||
|
||||
```
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
- Python ≥ 3.10,**NumPy + Matplotlib**,不需要 SciPy。
|
||||
- 树莓派上采集还需要 `picamera2`(系统包)和 `python3-tk`(交互窗口)。
|
||||
- 在 PC 上分析已有录制只需要 NumPy + Matplotlib。
|
||||
|
||||
## 快速开始
|
||||
|
||||
录制(树莓派):
|
||||
|
||||
```bash
|
||||
python3 hq_record.py --output recordings --exposure 66351 --gain 16 --duration 10
|
||||
```
|
||||
|
||||
回放 / 看谱(PC 或树莓派):
|
||||
|
||||
```bash
|
||||
python3 hq_spectrum.py --input recordings/hq_20261101_114417 --fps 15
|
||||
python3 wavelength_player.py --input recordings/hq_20261101_114417 \
|
||||
--calibration wavelength_calibration/calibration.json
|
||||
```
|
||||
|
||||
寻峰并导出(`--calibration` 只是**应用**已有的标定,不给就只输出像素坐标):
|
||||
|
||||
```bash
|
||||
python peak_finder.py --input recordings/hq_20261101_114417 \
|
||||
--calibration wavelength_calibration/calibration.json --select all
|
||||
python peak_finder.py --input recordings/hq_20261101_114417 --pick
|
||||
```
|
||||
|
||||
重新标定:
|
||||
|
||||
```bash
|
||||
python wavelength_calibration.py --input recordings/hq_20261101_114417 --headless
|
||||
# 看清 candidates.png 后指定:候选编号:波长
|
||||
python wavelength_calibration.py --input recordings/hq_20261101_114417 --headless \
|
||||
--assign "B1:404.6563,B2:435.8328,G1:546.0735,G2:576.9598,G3:579.0663"
|
||||
```
|
||||
|
||||
## 测试
|
||||
|
||||
```bash
|
||||
python -m unittest test_peak_finder test_wavelength_player -v
|
||||
```
|
||||
|
||||
端到端用例需要录制与标定;录制不在脚本附近时用环境变量指过去,否则自动跳过:
|
||||
|
||||
```powershell
|
||||
$env:PEAK_FINDER_TEST_RECORDING = "C:\Users\peng0\Dev\Light\recordings\hq_20261101_114417"
|
||||
$env:PEAK_FINDER_TEST_CALIBRATION = "wavelength_calibration\calibration.json"
|
||||
```
|
||||
|
||||
## 仓库里不放什么
|
||||
|
||||
`recordings/`(原始录制,单个约 470 MB)、`peak_exports/`(导出结果)、`__pycache__/`
|
||||
都在 `.gitignore` 里。标定结果 `wavelength_calibration/` 是**放**进来的,因为它决定了
|
||||
所有波长读数。
|
||||
@@ -0,0 +1,29 @@
|
||||
# 使用标定结果播放录制
|
||||
|
||||
将 `wavelength_player.py` 放在 `hq_spectrum.py` 同目录,使用原来的 Python/NumPy/Matplotlib 环境。
|
||||
原播放器不需要修改。默认标定路径是新脚本旁 `wavelength_calibration/calibration.json`。
|
||||
|
||||
在 Programs 目录运行,把下面的 `other_recording` 换成实际录制目录:
|
||||
|
||||
```powershell
|
||||
python wavelength_player.py --input recordings/other_recording --calibration wavelength_calibration/calibration.json
|
||||
```
|
||||
|
||||
默认暂停,沿用原播放器的播放/暂停、进度条、逐帧、倍速、循环、ROI 框选、`s` 保存和 `q` 退出。加 `--autoplay` 立即播放。默认在目标录制中自动检测 ROI;也可使用 `--roi calibration` 复用标定 ROI,或 `--roi 400,600,1400,900` 指定新 ROI。
|
||||
|
||||
```powershell
|
||||
# 用已有汞灯录制检查标定回放
|
||||
python wavelength_player.py --input recordings/hq_20261024_135709 --roi calibration
|
||||
# 无窗口检查前两帧并导出最后一帧截图及光谱 CSV
|
||||
python wavelength_player.py --input recordings/other_recording --headless --frames 2 --save-prefix replay_check
|
||||
# 全部参数
|
||||
python wavelength_player.py --help
|
||||
```
|
||||
|
||||
直接使用 JSON 中的降幂系数,不重新拟合。R/G/B 各自完整传感器列坐标转换为 nm,ROI 改动不会把像素坐标重新从零开始。线性、二次及波长递减方向均支持;瀑布图显示时重采样到等距波长网格,导出的 R/G/B 光谱仍保留各自原始采样坐标。绿色背景表示确认点覆盖的波长范围,范围外为外推。
|
||||
|
||||
默认不平滑、不降噪、不扣背景,保留包括负值的黑电平校正信号。各帧采用录制自己的黑电平元数据,不沿用标定灯录制的 4096;可用 `--black` 手动覆盖(单位为解码后的 ADU)。归一化副本仅用于寻峰和辅助显示。导出 CSV 沿用原播放器字段,其中 `signal_adu` 和 `signal_smoothed` 在本程序中相同,均未经平滑。强度单位仍为相机 ADU,未做光谱响应校正。
|
||||
|
||||
标定要求光学结构、传感器模式及坐标几何保持一致。若 JSON 指向的原标定录制仍存在,会比较图像宽高,不一致即拒绝;若原录制已移走,会提示无法核对尺寸。相同宽高也不能保证光学位置未改变。
|
||||
|
||||
测试:`python -m unittest test_wavelength_player -v`。
|
||||
+176
@@ -0,0 +1,176 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Average hq_record recording directories using the existing player's decoder."""
|
||||
from __future__ import annotations
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import sys
|
||||
import time
|
||||
import uuid
|
||||
|
||||
import numpy as np
|
||||
|
||||
try: # shared core lives beside this script
|
||||
import spectro_core as core
|
||||
except ImportError as exc: # pragma: no cover - placement error
|
||||
raise SystemExit('hq_average.py needs spectro_core.py in the same directory '
|
||||
'(or on sys.path)') from exc
|
||||
|
||||
|
||||
def load_player(source, module_dir=None):
|
||||
"""Decoder module: --module-dir, then this folder, then the project above."""
|
||||
for directory in ([Path(module_dir)] if module_dir
|
||||
else [Path(__file__).resolve().parent, Path(source).parent.parent]):
|
||||
if (directory / 'hq_spectrum.py').is_file():
|
||||
return core.load_hq(directory)
|
||||
return core.load_hq()
|
||||
|
||||
|
||||
def write_json(path, value):
|
||||
core.write_json(path, value)
|
||||
|
||||
|
||||
def average(source, output, group=0, drop_last=False, module_dir=None):
|
||||
source, output = Path(source).resolve(), Path(output).resolve()
|
||||
if group < 0:
|
||||
raise ValueError('group must be >= 0')
|
||||
if source == output or source in output.parents or output in source.parents:
|
||||
raise ValueError('Input and output must be separate, non-nested directories')
|
||||
if output.exists():
|
||||
raise ValueError(f'Output already exists: {output}')
|
||||
meta = json.loads((source / 'meta.json').read_text(encoding='utf-8'))
|
||||
if meta.get('kind') != 'hq_raw_recording' or meta.get('format_version') != 1:
|
||||
raise ValueError('Expected hq_raw_recording format_version 1')
|
||||
w, h, stride = (int(meta[k]) for k in ('width', 'height', 'stride'))
|
||||
fb = int(meta['frame_bytes'])
|
||||
if min(w, h, stride) <= 0 or fb != h * stride:
|
||||
raise ValueError('Invalid dimensions/stride/frame_bytes')
|
||||
size = (source / 'frames.bin').stat().st_size
|
||||
count, remainder = divmod(size, fb)
|
||||
if remainder or not count or int(meta.get('frames', count)) != count:
|
||||
raise ValueError('Truncated recording or frame count mismatch; repair source separately')
|
||||
if not meta.get('completed'):
|
||||
raise ValueError('Source recording is not marked completed')
|
||||
records = []
|
||||
metadata_path = source / 'frames.jsonl'
|
||||
if metadata_path.exists():
|
||||
for i, line in enumerate(metadata_path.read_text(encoding='utf-8').splitlines()):
|
||||
row = json.loads(line)
|
||||
if not isinstance(row, dict):
|
||||
raise ValueError(f'Invalid metadata row {i}')
|
||||
nested = row.pop('metadata', None)
|
||||
if isinstance(nested, dict):
|
||||
row = {**nested, **row}
|
||||
for key, expected in [('i', i), ('offset', i * fb), ('bytes', fb)]:
|
||||
if key in row and row[key] != expected:
|
||||
raise ValueError(f'Incorrect metadata {key} in row {i}')
|
||||
records.append(row)
|
||||
if len(records) != count:
|
||||
raise ValueError('frames.jsonl count mismatch')
|
||||
else:
|
||||
records = [{} for _ in range(count)]
|
||||
# Exposure/gain changes are retained as ranges in each averaged frame's provenance.
|
||||
player = load_player(source, module_dir)
|
||||
pattern = str(meta.get('pattern') or player.pattern_from_format(meta['raw_format']))
|
||||
if pattern not in ('BGGR', 'RGGB', 'GBRG', 'GRBG'):
|
||||
raise ValueError('Unsupported Bayer pattern')
|
||||
config = {'size': (w, h), 'stride': stride}
|
||||
n = group or count
|
||||
groups = [(a, min(a + n, count)) for a in range(0, count, n)
|
||||
if not (drop_last and min(a + n, count) - a < n)]
|
||||
if not groups:
|
||||
raise ValueError('No output frames; reduce group size or omit --drop-last')
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
temp = output.parent / (output.name + '.partial-' + uuid.uuid4().hex[:8])
|
||||
temp.mkdir()
|
||||
started = time.monotonic()
|
||||
try:
|
||||
depth = None
|
||||
out_fb = w * h * 2
|
||||
with (source / 'frames.bin').open('rb') as inp, (temp / 'frames.bin').open('wb') as raw_out, (temp / 'frames.jsonl').open('w', encoding='utf-8') as mout:
|
||||
for oi, (a, b) in enumerate(groups):
|
||||
acc = np.zeros((h, w), dtype=np.float64)
|
||||
inp.seek(a * fb)
|
||||
for index in range(a, b):
|
||||
data = inp.read(fb)
|
||||
if len(data) != fb:
|
||||
raise ValueError('Input changed or was truncated while reading')
|
||||
image, bits = player.decode_raw(np.frombuffer(data, dtype=np.uint8).reshape(h, stride), config, meta['raw_format'])
|
||||
if depth is not None and depth != bits:
|
||||
raise ValueError('Decoded bit depth changed')
|
||||
depth = bits
|
||||
acc += image
|
||||
image = np.clip(np.rint(acc / (b - a)), 0, (1 << depth) - 1).astype('<u2')
|
||||
raw_out.write(image.tobytes())
|
||||
row = dict(records[a])
|
||||
for key in ('_frame_index', '_frame_total'):
|
||||
row.pop(key, None)
|
||||
ranges = {}
|
||||
for key in ('ExposureTime', 'AnalogueGain', 'DigitalGain'):
|
||||
values = [r[key] for r in records[a:b] if isinstance(r.get(key), (float, int))]
|
||||
if values:
|
||||
ranges[key] = [min(values), max(values)]
|
||||
row.update(i=oi, offset=oi * out_fb, bytes=out_fb,
|
||||
averaging={'start_index': a, 'end_index_exclusive': b, 'count': b-a,
|
||||
'metadata_policy': 'representative first source frame',
|
||||
'last_source_metadata': records[b-1], 'parameter_ranges': ranges})
|
||||
mout.write(json.dumps(row, ensure_ascii=False) + '\n')
|
||||
print(f'Output {oi+1}/{len(groups)}: source [{a}, {b})', flush=True)
|
||||
newmeta = dict(meta)
|
||||
fps = float(meta.get('average_capture_fps') or meta.get('fps_requested') or 15) / n
|
||||
newmeta.update(raw_format=f'S{pattern}{depth}', bit_depth=depth, stride=w*2,
|
||||
frame_bytes=out_fb, storage_dtype='uint8', frames=len(groups),
|
||||
completed=True, frames_file='frames.bin', metadata_file='frames.jsonl',
|
||||
average_capture_fps=fps, fps_requested=fps,
|
||||
duration_s=len(groups)/fps, frame_limit=len(groups), duration_limit_s=0,
|
||||
averaging={'source_directory': str(source), 'source_meta': meta,
|
||||
'mode': 'all' if not group else 'group', 'group_size': n,
|
||||
'drop_last': drop_last, 'discarded_frames': count-groups[-1][1],
|
||||
'rounding': 'nearest, ties to even', 'accumulator': 'float64',
|
||||
'timing': 'nominal group cadence; last partial group may be shorter'})
|
||||
write_json(temp / 'meta.json', newmeta)
|
||||
# Keep every original per-frame field, including within-group variations.
|
||||
if metadata_path.exists():
|
||||
import shutil
|
||||
shutil.copyfile(metadata_path, temp / 'source_frames.jsonl')
|
||||
os.rename(temp, output)
|
||||
print(f'Saved: {output} ({time.monotonic()-started:.1f}s)')
|
||||
return output
|
||||
except BaseException:
|
||||
print(f'Incomplete output kept for inspection: {temp}', file=sys.stderr)
|
||||
raise
|
||||
|
||||
|
||||
def main():
|
||||
p = argparse.ArgumentParser(description=__doc__)
|
||||
p.add_argument('--input', type=Path, help='Recording directory; omit for folder picker')
|
||||
p.add_argument('--output', type=Path, help='NEW output directory')
|
||||
m = p.add_mutually_exclusive_group()
|
||||
m.add_argument('--all', action='store_true', help='Average entire recording (default)')
|
||||
m.add_argument('--group', type=int, default=0, help='Average each N consecutive frames')
|
||||
p.add_argument('--drop-last', action='store_true', help='Discard final incomplete group')
|
||||
p.add_argument('--module-dir', type=Path, help='Directory containing original hq_spectrum.py')
|
||||
args = p.parse_args()
|
||||
if args.input is None:
|
||||
import tkinter as tk
|
||||
from tkinter import filedialog
|
||||
root = tk.Tk()
|
||||
root.withdraw()
|
||||
selected = filedialog.askdirectory(title='Select recording folder')
|
||||
root.destroy()
|
||||
if not selected:
|
||||
return 0
|
||||
args.input = Path(selected)
|
||||
suffix = '_avg_all' if not args.group else f'_avg{args.group}'
|
||||
output = args.output or args.input.resolve().with_name(args.input.name + suffix)
|
||||
try:
|
||||
average(args.input, output, args.group, args.drop_last, args.module_dir)
|
||||
return 0
|
||||
except (Exception, KeyboardInterrupt) as exc:
|
||||
print(f'Error: {exc}', file=sys.stderr)
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
raise SystemExit(main())
|
||||
+396
@@ -0,0 +1,396 @@
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
hq_record.py
|
||||
============
|
||||
Fast raw recorder for the Raspberry Pi HQ Camera.
|
||||
|
||||
This is the "record" half of the split workflow:
|
||||
|
||||
hq_record.py -> recordings/hq_YYYYmmdd_HHMMSS/
|
||||
hq_spectrum.py --input REC_DIR -> replay / analysis
|
||||
|
||||
The recorder does not import Matplotlib and does not compute spectra. It only
|
||||
pulls the Picamera2 RAW stream, queues each raw byte buffer, and writes it to
|
||||
`frames.bin`. A background writer thread keeps disk I/O away from the camera
|
||||
capture loop.
|
||||
|
||||
Recording layout
|
||||
----------------
|
||||
REC_DIR/
|
||||
meta.json camera/format/stride/frame count
|
||||
frames.bin raw byte buffers, one after another, no decoding
|
||||
frames.jsonl one line of camera metadata per frame
|
||||
|
||||
For the usual Pi 5 HQ Camera configuration the raw stream is
|
||||
`BGGR_PISP_COMP1` (PC1B). Each frame is height*stride bytes; at 2028x1520
|
||||
that is typically about 3.1 MB/frame. At 15 fps that is about 47 MB/s, so a
|
||||
USB SSD/NVMe is preferable to a slow SD card.
|
||||
|
||||
Examples
|
||||
--------
|
||||
# list modes first if needed
|
||||
python3 hq_record.py --list-modes
|
||||
|
||||
# 10 seconds, 15 fps, manual exposure copied from rpicam-hello
|
||||
python3 hq_record.py --duration 10 --fps 15 \
|
||||
--exposure 33013 --gain 16 \
|
||||
--record-dir recordings/hg_lamp
|
||||
|
||||
# unlimited until Ctrl+C
|
||||
python3 hq_record.py --duration 0 --fps 15 \
|
||||
--record-dir recordings/hg_long
|
||||
|
||||
# analyze later (on the Pi or on a PC)
|
||||
python3 hq_spectrum.py --input recordings/hg_lamp --fps 15
|
||||
|
||||
The recorder is deliberately independent of the analyzer: a slow Matplotlib
|
||||
redraw can no longer cause dropped frames or change the exposure timing.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import queue
|
||||
import shutil
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
from typing import Optional
|
||||
|
||||
import numpy as np
|
||||
|
||||
try:
|
||||
from hq_spectrum import CameraSource, list_camera_modes
|
||||
except ImportError as exc: # pragma: no cover - import error path
|
||||
raise SystemExit(
|
||||
"hq_record.py must sit next to hq_spectrum.py "
|
||||
"(run it from the hq_spectrum directory)."
|
||||
) from exc
|
||||
|
||||
|
||||
class RawWriter(threading.Thread):
|
||||
"""Write queued raw buffers and metadata lines in the background."""
|
||||
|
||||
def __init__(self, frames_path: Path, metadata_path: Path, frame_bytes: int,
|
||||
queue_size: int):
|
||||
super().__init__(name="hq-raw-writer", daemon=False)
|
||||
self.frames_path = frames_path
|
||||
self.metadata_path = metadata_path
|
||||
self.frame_bytes = frame_bytes
|
||||
self.queue: queue.Queue = queue.Queue(maxsize=max(2, queue_size))
|
||||
self.frames_written = 0
|
||||
self.error: Optional[BaseException] = None
|
||||
self.finished = threading.Event()
|
||||
|
||||
def put(self, raw: np.ndarray, metadata: dict, host_time_ns: int) -> None:
|
||||
if self.error is not None:
|
||||
raise RuntimeError(f"writer failed: {self.error}")
|
||||
raw = np.ascontiguousarray(raw)
|
||||
if raw.dtype != np.uint8 or raw.nbytes != self.frame_bytes:
|
||||
raise ValueError(
|
||||
f"unexpected raw buffer {raw.shape} {raw.dtype} {raw.nbytes}B; "
|
||||
f"expected {self.frame_bytes}B uint8"
|
||||
)
|
||||
item = (raw, dict(metadata), int(host_time_ns))
|
||||
while True:
|
||||
try:
|
||||
self.queue.put(item, timeout=0.5)
|
||||
return
|
||||
except queue.Full:
|
||||
if self.error is not None:
|
||||
raise RuntimeError(f"writer failed: {self.error}")
|
||||
|
||||
def run(self) -> None:
|
||||
try:
|
||||
with open(self.frames_path, "wb", buffering=4 * 1024 * 1024) as frames, \
|
||||
open(self.metadata_path, "w", encoding="utf-8", buffering=1024 * 1024) as meta_out:
|
||||
while True:
|
||||
item = self.queue.get()
|
||||
if item is None:
|
||||
break
|
||||
raw, camera_meta, host_time_ns = item
|
||||
offset = self.frames_written * self.frame_bytes
|
||||
record = dict(camera_meta)
|
||||
record.update({
|
||||
"i": self.frames_written,
|
||||
"offset": offset,
|
||||
"bytes": raw.nbytes,
|
||||
"host_time_ns": host_time_ns,
|
||||
})
|
||||
frames.write(memoryview(raw).cast("B"))
|
||||
meta_out.write(json.dumps(record, default=str, separators=(",", ":")) + "\n")
|
||||
self.frames_written += 1
|
||||
if self.frames_written % 30 == 0:
|
||||
frames.flush()
|
||||
meta_out.flush()
|
||||
frames.flush()
|
||||
meta_out.flush()
|
||||
self.finished.set()
|
||||
except BaseException as exc: # noqa: BLE001 - report writer failure to main
|
||||
self.error = exc
|
||||
self.finished.set()
|
||||
|
||||
def stop(self) -> None:
|
||||
if self.is_alive():
|
||||
try:
|
||||
self.queue.put(None, timeout=2.0)
|
||||
except queue.Full:
|
||||
pass
|
||||
self.join(timeout=30.0)
|
||||
if self.is_alive():
|
||||
raise RuntimeError("writer thread did not stop within 30 seconds")
|
||||
if self.error is not None:
|
||||
raise RuntimeError(f"writer failed: {self.error}")
|
||||
|
||||
|
||||
def parse_args(argv: Optional[list[str]] = None) -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Record Raspberry Pi HQ Camera RAW frames for later spectrum analysis.",
|
||||
formatter_class=argparse.ArgumentDefaultsHelpFormatter,
|
||||
)
|
||||
parser.add_argument("--width", type=int, default=2028, help="raw stream width")
|
||||
parser.add_argument("--height", type=int, default=1520, help="raw stream height")
|
||||
parser.add_argument("--fps", type=float, default=15.0, help="requested frame rate")
|
||||
parser.add_argument("--duration", type=float, default=10.0,
|
||||
help="seconds to record; 0 means until Ctrl+C")
|
||||
parser.add_argument("--frames", type=int, default=0,
|
||||
help="optional frame limit; 0 means no limit")
|
||||
parser.add_argument("--record-dir", default=None,
|
||||
help="output directory (default recordings/hq_YYYYmmdd_HHMMSS)")
|
||||
parser.add_argument("--queue-size", type=int, default=8,
|
||||
help="raw frames buffered in RAM while the writer catches up")
|
||||
parser.add_argument("--exposure", type=float, default=None,
|
||||
help="manual exposure time in microseconds")
|
||||
parser.add_argument("--gain", type=float, default=None, help="manual analogue gain")
|
||||
parser.add_argument("--raw-format", default=None,
|
||||
help="optional raw format, e.g. SBGGR12 or BGGR_PISP_COMP1")
|
||||
parser.add_argument("--list-modes", action="store_true",
|
||||
help="list camera raw modes and exit")
|
||||
args = parser.parse_args(argv)
|
||||
if args.width < 64 or args.height < 64:
|
||||
parser.error("width and height must be at least 64")
|
||||
if args.fps <= 0:
|
||||
parser.error("--fps must be positive")
|
||||
if args.duration < 0:
|
||||
parser.error("--duration must be >= 0")
|
||||
if args.frames < 0:
|
||||
parser.error("--frames must be >= 0")
|
||||
if args.queue_size < 2:
|
||||
parser.error("--queue-size must be >= 2")
|
||||
return args
|
||||
|
||||
|
||||
def write_json(path: Path, data: dict) -> None:
|
||||
tmp = path.with_suffix(path.suffix + ".tmp")
|
||||
with open(tmp, "w", encoding="utf-8") as handle:
|
||||
json.dump(data, handle, indent=2, default=str)
|
||||
handle.write("\n")
|
||||
tmp.replace(path)
|
||||
|
||||
|
||||
def prepare_directory(path: Path) -> None:
|
||||
if path.exists():
|
||||
if not path.is_dir():
|
||||
raise RuntimeError(f"{path} exists and is not a directory")
|
||||
if any(path.iterdir()):
|
||||
raise RuntimeError(
|
||||
f"{path} is not empty; choose a new --record-dir "
|
||||
f"(the recorder never overwrites old data)"
|
||||
)
|
||||
else:
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
|
||||
def main(argv: Optional[list[str]] = None) -> int:
|
||||
args = parse_args(argv)
|
||||
if args.list_modes:
|
||||
list_camera_modes()
|
||||
return 0
|
||||
|
||||
cam_args = SimpleNamespace(
|
||||
width=args.width,
|
||||
height=args.height,
|
||||
fps=args.fps,
|
||||
exposure=args.exposure,
|
||||
gain=args.gain,
|
||||
raw_format=args.raw_format,
|
||||
)
|
||||
|
||||
source = None
|
||||
writer = None
|
||||
meta = None
|
||||
record_dir = None
|
||||
enqueued = 0
|
||||
started = None
|
||||
exit_code = 0
|
||||
|
||||
try:
|
||||
source = CameraSource(cam_args)
|
||||
print(f"camera : {source.description}")
|
||||
print(f"raw stream : {source.raw_format} "
|
||||
f"{source.full_size[0]}x{source.full_size[1]}")
|
||||
print(f"stride : {source.stride}")
|
||||
print(f"Bayer : {source.pattern} decoded {source.bit_depth}-bit buffer")
|
||||
print(f"target rate : {args.fps:g} fps")
|
||||
if "PISP_COMP" in source.raw_format.upper():
|
||||
print("note : recording the compressed PC1B buffer; no decode on the Pi")
|
||||
|
||||
source.start()
|
||||
last_meta = {}
|
||||
# Discard warm-up frames; the recording should start with settled AE.
|
||||
for _ in range(5):
|
||||
_, last_meta = source.read_raw()
|
||||
|
||||
# Get one sample only to validate the actual buffer size.
|
||||
sample_raw, sample_meta = source.read_raw()
|
||||
last_meta = sample_meta
|
||||
if sample_raw.dtype != np.uint8:
|
||||
raise RuntimeError(f"unexpected raw dtype {sample_raw.dtype}; expected uint8 bytes")
|
||||
frame_bytes = int(sample_raw.nbytes)
|
||||
expected_bytes = source.stride * source.full_size[1]
|
||||
if frame_bytes != expected_bytes:
|
||||
raise RuntimeError(
|
||||
f"raw buffer is {frame_bytes} B but stride*height is {expected_bytes} B; "
|
||||
f"refusing to write a recording that the analyzer would misread"
|
||||
)
|
||||
|
||||
stamp = datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||
record_dir = Path(args.record_dir).expanduser() if args.record_dir else \
|
||||
Path("recordings") / f"hq_{stamp}"
|
||||
prepare_directory(record_dir)
|
||||
frames_path = record_dir / "frames.bin"
|
||||
metadata_path = record_dir / "frames.jsonl"
|
||||
meta_path = record_dir / "meta.json"
|
||||
|
||||
data_rate = frame_bytes * args.fps
|
||||
free = shutil.disk_usage(record_dir).free
|
||||
print(f"output : {record_dir}")
|
||||
print(f"frame size : {frame_bytes / 1e6:.3f} MB")
|
||||
print(f"data rate : {data_rate / 1e6:.1f} MB/s")
|
||||
print(f"free space : {free / 1e9:.2f} GB")
|
||||
planned_frames = None
|
||||
if args.frames:
|
||||
planned_frames = args.frames
|
||||
if args.duration:
|
||||
duration_frames = int(args.duration * args.fps)
|
||||
planned_frames = duration_frames if planned_frames is None else min(planned_frames, duration_frames)
|
||||
if planned_frames is not None:
|
||||
needed = frame_bytes * planned_frames * 1.05
|
||||
print(f"planned size : {needed / 1e9:.2f} GB for ~{planned_frames} frames")
|
||||
if needed > free:
|
||||
raise RuntimeError(
|
||||
f"not enough free space: need ~{needed / 1e9:.2f} GB, "
|
||||
f"have {free / 1e9:.2f} GB"
|
||||
)
|
||||
|
||||
meta = {
|
||||
"format_version": 1,
|
||||
"kind": "hq_raw_recording",
|
||||
"created_at": datetime.now().astimezone().isoformat(timespec="seconds"),
|
||||
"camera": source.description,
|
||||
"raw_format": source.raw_format,
|
||||
"width": source.full_size[0],
|
||||
"height": source.full_size[1],
|
||||
"stride": source.stride,
|
||||
"frame_bytes": frame_bytes,
|
||||
"storage_dtype": "uint8",
|
||||
"pattern": source.pattern,
|
||||
"bit_depth": source.bit_depth,
|
||||
"fps_requested": args.fps,
|
||||
"exposure_us": args.exposure,
|
||||
"gain": args.gain,
|
||||
"black_level": last_meta.get("BlackLevel"),
|
||||
"duration_limit_s": args.duration,
|
||||
"frame_limit": args.frames,
|
||||
"frames": 0,
|
||||
"completed": False,
|
||||
}
|
||||
write_json(meta_path, meta)
|
||||
|
||||
writer = RawWriter(frames_path, metadata_path, frame_bytes, args.queue_size)
|
||||
writer.start()
|
||||
# The first sample was taken to validate the format, so it is also the
|
||||
# first frame of the recording.
|
||||
writer.put(sample_raw, last_meta, time.time_ns())
|
||||
enqueued = 1
|
||||
|
||||
started = time.monotonic()
|
||||
last_report = started
|
||||
while True:
|
||||
if args.frames and enqueued >= args.frames:
|
||||
break
|
||||
elapsed = time.monotonic() - started
|
||||
if args.duration and elapsed >= args.duration:
|
||||
break
|
||||
raw, camera_meta = source.read_raw()
|
||||
writer.put(raw, camera_meta, time.time_ns())
|
||||
enqueued += 1
|
||||
now = time.monotonic()
|
||||
if now - last_report >= 1.0:
|
||||
written = writer.frames_written
|
||||
rate = enqueued / max(now - started, 1e-6)
|
||||
print(f"recording : {enqueued} frames captured, {written} written, "
|
||||
f"{rate:.1f} fps")
|
||||
last_report = now
|
||||
|
||||
elapsed = time.monotonic() - started
|
||||
writer.stop()
|
||||
writer_error = writer.error
|
||||
writer = None
|
||||
if writer_error is not None:
|
||||
raise RuntimeError(f"writer failed: {writer_error}")
|
||||
|
||||
meta.update({
|
||||
"frames": int(enqueued),
|
||||
"duration_s": round(elapsed, 3),
|
||||
"average_capture_fps": round(enqueued / max(elapsed, 1e-6), 3),
|
||||
"completed": True,
|
||||
"ended_at": datetime.now().astimezone().isoformat(timespec="seconds"),
|
||||
"frames_file": "frames.bin",
|
||||
"metadata_file": "frames.jsonl",
|
||||
})
|
||||
write_json(meta_path, meta)
|
||||
size = frames_path.stat().st_size
|
||||
print(f"done : {enqueued} frames, {size / 1e9:.2f} GB, "
|
||||
f"{enqueued / max(elapsed, 1e-6):.1f} fps")
|
||||
print(f"analyze with : python3 hq_spectrum.py --input \"{record_dir}\"")
|
||||
return 0
|
||||
except KeyboardInterrupt:
|
||||
print("\ninterrupted; finalizing recording", file=sys.stderr)
|
||||
exit_code = 130
|
||||
return exit_code
|
||||
except Exception as exc:
|
||||
print(f"error: {exc}", file=sys.stderr)
|
||||
exit_code = 1
|
||||
return exit_code
|
||||
finally:
|
||||
if source is not None:
|
||||
try:
|
||||
source.close()
|
||||
except Exception:
|
||||
pass
|
||||
if writer is not None:
|
||||
try:
|
||||
writer.stop()
|
||||
except Exception as exc:
|
||||
print(f"writer shutdown error: {exc}", file=sys.stderr)
|
||||
exit_code = 1
|
||||
if meta is not None and record_dir is not None:
|
||||
try:
|
||||
meta.update({
|
||||
"frames": int(writer.frames_written) if writer is not None else int(enqueued),
|
||||
"completed": exit_code == 0 and meta.get("completed", False),
|
||||
"ended_at": datetime.now().astimezone().isoformat(timespec="seconds"),
|
||||
})
|
||||
write_json(record_dir / "meta.json", meta)
|
||||
except Exception as exc:
|
||||
print(f"warning: could not finalize meta.json: {exc}", file=sys.stderr)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
+2409
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,6 @@
|
||||
wavelength_nm,species,label
|
||||
404.6563,Hg I,Hg 405
|
||||
435.8328,Hg I,Hg 436
|
||||
546.0735,Hg I,Hg 546
|
||||
576.9598,Hg I,Hg 577 (unresolved with 579 in this instrument)
|
||||
579.0663,Hg I,Hg 579
|
||||
|
+1411
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,6 @@
|
||||
# 分析端(PC)依赖:读录制、寻峰、画图、标定
|
||||
numpy>=1.24
|
||||
matplotlib>=3.6
|
||||
|
||||
# 树莓派采集端还需要(系统包,不走 pip;按需安装):
|
||||
# sudo apt install python3-picamera2 python3-numpy python3-matplotlib python3-tk
|
||||
+860
@@ -0,0 +1,860 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Shared core for the spectro tools: loading, calibration, peak detection, output.
|
||||
|
||||
Extracted from peak_finder.py so that wavelength_calibration.py, wavelength_player.py
|
||||
and hq_average.py can call the same code instead of keeping their own copies.
|
||||
|
||||
No command line and no matplotlib at import time; plotting helpers stay in the
|
||||
scripts that need them. Everything here is pure NumPy.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import importlib.util
|
||||
import json
|
||||
import math
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
import numpy as np
|
||||
|
||||
CORE_DIR = Path(__file__).resolve().parent
|
||||
GAUSS_TO_FWHM = 2.3548200450309493
|
||||
|
||||
|
||||
def load_hq(module_dir: Optional[str] = None) -> object:
|
||||
"""Import hq_spectrum.py from this folder (or ``module_dir``) by path."""
|
||||
|
||||
candidates = [Path(module_dir).expanduser()] if module_dir else [CORE_DIR, Path.cwd()]
|
||||
for directory in candidates:
|
||||
path = directory / "hq_spectrum.py"
|
||||
if path.is_file():
|
||||
spec = importlib.util.spec_from_file_location("hq_spectrum_shared", path)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
sys.modules[spec.name] = module
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
try: # last resort: a normal import from sys.path
|
||||
import hq_spectrum as module # type: ignore
|
||||
except ImportError as exc:
|
||||
raise RuntimeError("cannot find hq_spectrum.py; put it beside spectro_core.py "
|
||||
"or use --module-dir") from exc
|
||||
return module
|
||||
|
||||
|
||||
def is_saturated(image: np.ndarray, roi, x_center: float, full_scale: float,
|
||||
band: tuple[int, int] = (-2, 4), threshold: float = 0.01) -> bool:
|
||||
"""True when a meaningful fraction of the pixels near ``x_center`` is clipped."""
|
||||
|
||||
x0 = max(0, min(int(x_center) + band[0], image.shape[1]))
|
||||
x1 = max(0, min(int(x_center) + band[1], image.shape[1]))
|
||||
block = image[roi.y0:roi.y1, x0:x1]
|
||||
if block.size == 0:
|
||||
return False
|
||||
return float(np.mean(block >= 0.995 * full_scale)) > threshold
|
||||
|
||||
|
||||
def _num(value, digits: int = 6) -> str:
|
||||
"""Format a value for CSV output; non-finite values become empty cells."""
|
||||
|
||||
if value is None:
|
||||
return ""
|
||||
if isinstance(value, (bool, np.bool_)):
|
||||
return "1" if value else "0"
|
||||
if isinstance(value, str):
|
||||
return value
|
||||
try:
|
||||
number = float(value)
|
||||
except (TypeError, ValueError):
|
||||
return str(value)
|
||||
if not np.isfinite(number):
|
||||
return ""
|
||||
return f"{number:.{digits}g}"
|
||||
|
||||
|
||||
def write_csv(path: Path, fields: list[str], rows) -> None:
|
||||
with path.open("w", newline="", encoding="utf-8") as handle:
|
||||
writer = csv.writer(handle)
|
||||
writer.writerow(fields)
|
||||
for row in rows:
|
||||
writer.writerow(row)
|
||||
|
||||
|
||||
def _jsonable(value):
|
||||
"""Convert NumPy scalars/arrays and Paths into plain JSON values."""
|
||||
|
||||
if isinstance(value, dict):
|
||||
return {str(k): _jsonable(v) for k, v in value.items()}
|
||||
if isinstance(value, (list, tuple)):
|
||||
return [_jsonable(v) for v in value]
|
||||
if isinstance(value, np.ndarray):
|
||||
return [_jsonable(v) for v in value.tolist()]
|
||||
if isinstance(value, (np.floating, np.integer, np.bool_)):
|
||||
return value.item()
|
||||
if isinstance(value, Path):
|
||||
return str(value)
|
||||
return value
|
||||
|
||||
|
||||
def write_json(path, value) -> None:
|
||||
"""Write JSON with NumPy/Path values converted and a trailing newline."""
|
||||
|
||||
Path(path).write_text(json.dumps(_jsonable(value), indent=2) + "\n", encoding="utf-8")
|
||||
|
||||
|
||||
class Calibration:
|
||||
"""Polynomial sensor-pixel -> wavelength saved by wavelength_calibration.py."""
|
||||
|
||||
def __init__(self, path, offset_nm: float = 0.0):
|
||||
self.path = Path(path).expanduser().resolve()
|
||||
try:
|
||||
self.data = json.loads(self.path.read_text(encoding="utf-8-sig"))
|
||||
except OSError as exc:
|
||||
raise ValueError(f"cannot read calibration {self.path}: {exc}") from exc
|
||||
except json.JSONDecodeError as exc:
|
||||
raise ValueError(f"calibration {self.path} is not valid JSON: {exc}") from exc
|
||||
if not isinstance(self.data, dict):
|
||||
raise ValueError("calibration JSON must contain an object")
|
||||
coeff = np.asarray(self.data.get("coefficients_descending"), dtype=float)
|
||||
if coeff.ndim != 1 or len(coeff) not in (2, 3):
|
||||
raise ValueError("calibration needs 2 or 3 descending coefficients")
|
||||
if not np.all(np.isfinite(coeff)) or coeff[0] == 0:
|
||||
raise ValueError("calibration coefficients must be finite and the leading one nonzero")
|
||||
if self.data.get("order", len(coeff) - 1) != len(coeff) - 1:
|
||||
raise ValueError("calibration order does not match coefficient count")
|
||||
self.coeff = coeff
|
||||
span = np.asarray(self.data.get("calibrated_pixel_range"), dtype=float)
|
||||
if span.shape != (2,) or not np.all(np.isfinite(span)) or span[0] >= span[1]:
|
||||
raise ValueError("calibration has an invalid calibrated_pixel_range")
|
||||
self.span = span
|
||||
self.offset_nm = float(offset_nm)
|
||||
if not np.isfinite(self.offset_nm):
|
||||
raise ValueError("wavelength offset must be finite")
|
||||
self.width: Optional[int] = None
|
||||
self._grid_px = None
|
||||
self._grid_nm = None
|
||||
|
||||
def __call__(self, x):
|
||||
return np.polyval(self.coeff, np.asarray(x, dtype=float)) + self.offset_nm
|
||||
|
||||
def dispersion(self, x) -> np.ndarray:
|
||||
"""Local nm per sensor pixel (never zero over a validated sensor)."""
|
||||
|
||||
return np.polyval(np.polyder(self.coeff), np.asarray(x, dtype=float))
|
||||
|
||||
def set_width(self, width: int) -> None:
|
||||
self.width = int(width)
|
||||
grid = np.arange(self.width, dtype=float)
|
||||
values = np.asarray(self(grid), dtype=float)
|
||||
order = np.argsort(values)
|
||||
self._grid_px = grid[order]
|
||||
self._grid_nm = values[order]
|
||||
|
||||
def inverse(self, wavelength):
|
||||
"""Sensor pixel of a wavelength; used for the secondary plot axis only."""
|
||||
|
||||
if self._grid_nm is None:
|
||||
raise ValueError("calibration width is unknown")
|
||||
return np.interp(np.asarray(wavelength, dtype=float), self._grid_nm, self._grid_px)
|
||||
|
||||
@property
|
||||
def nm_span(self) -> tuple[float, float]:
|
||||
values = sorted(np.asarray(self(self.span), dtype=float).tolist())
|
||||
return float(values[0]), float(values[1])
|
||||
|
||||
def describe(self) -> str:
|
||||
low, high = self.nm_span
|
||||
text = (f"JSON calibration ({self.data.get('wavelength_medium', 'air')}); "
|
||||
f"confirmed {low:.2f}-{high:.2f} nm; outside = extrapolation")
|
||||
if self.offset_nm:
|
||||
text += f"; manual offset {self.offset_nm:+.3f} nm"
|
||||
return text
|
||||
|
||||
def validate_geometry(self, source, tell=None) -> None:
|
||||
"""Refuse a recording whose sensor geometry differs from the calibration."""
|
||||
|
||||
tell = tell or (lambda *args, **kwargs: None)
|
||||
derivative = self.dispersion([0.0, float(source.full_size[0] - 1)])
|
||||
if np.any(np.abs(derivative) < 1e-12) or derivative[0] * derivative[1] <= 0:
|
||||
raise ValueError("calibration is not monotonic over this recording width")
|
||||
reference = self.data.get("recording")
|
||||
metadata = Path(reference) / "meta.json" if reference else None
|
||||
if metadata is not None and metadata.is_file():
|
||||
reference_meta = json.loads(metadata.read_text(encoding="utf-8-sig"))
|
||||
size = (int(reference_meta["width"]), int(reference_meta["height"]))
|
||||
if size != tuple(source.full_size):
|
||||
raise ValueError("recording dimensions differ from the calibration recording; "
|
||||
"recalibrate this sensor mode")
|
||||
else:
|
||||
tell("note : calibration reference recording unavailable; sensor size not checked")
|
||||
if self.data.get("pattern") and self.data["pattern"] != source.pattern:
|
||||
tell(f"note : calibration pattern {self.data['pattern']} differs from "
|
||||
f"recording pattern {source.pattern}")
|
||||
|
||||
|
||||
def _rank_filter(y: np.ndarray, width: int, reducer) -> np.ndarray:
|
||||
"""Running min/max over the window [i - width, i + width] with edge padding."""
|
||||
|
||||
width = max(0, int(width))
|
||||
values = np.asarray(y, dtype=float)
|
||||
if width == 0 or values.size <= 1:
|
||||
return values.copy()
|
||||
n = values.size
|
||||
padded = np.pad(values, (width, width), mode="edge")
|
||||
out = padded[width:width + n].copy()
|
||||
for shift in range(-width, width + 1):
|
||||
start = width + shift
|
||||
reducer(out, padded[start:start + n], out=out)
|
||||
return out
|
||||
|
||||
|
||||
def _min_filter(y: np.ndarray, width: int) -> np.ndarray:
|
||||
return _rank_filter(y, width, np.minimum)
|
||||
|
||||
|
||||
def _max_filter(y: np.ndarray, width: int) -> np.ndarray:
|
||||
return _rank_filter(y, width, np.maximum)
|
||||
|
||||
|
||||
def estimate_baseline(y: np.ndarray, window_samples: int, smooth) -> np.ndarray:
|
||||
"""Morphological opening baseline; removes peaks narrower than the window."""
|
||||
|
||||
width = max(0, int(window_samples))
|
||||
if width == 0 or np.size(y) <= 2:
|
||||
return np.zeros_like(np.asarray(y, dtype=float))
|
||||
opened = _max_filter(_min_filter(np.asarray(y, dtype=float), width), width)
|
||||
return smooth(opened.astype(np.float32), max(1, width // 2)).astype(float)
|
||||
|
||||
|
||||
def estimate_noise(net: np.ndarray) -> float:
|
||||
"""Robust noise proxy from the median absolute first difference."""
|
||||
|
||||
if np.size(net) < 3:
|
||||
return 0.0
|
||||
diff = np.diff(np.asarray(net, dtype=float))
|
||||
mad = float(np.median(np.abs(diff - np.median(diff))))
|
||||
return mad / (0.6745 * math.sqrt(2.0))
|
||||
|
||||
|
||||
def parabola_center(x: np.ndarray, y: np.ndarray, index: int) -> float:
|
||||
"""Sub-sample vertex of the parabola through three samples (NaN if unusable)."""
|
||||
|
||||
if index <= 0 or index >= y.size - 1:
|
||||
return float("nan")
|
||||
y0, y1, y2 = float(y[index - 1]), float(y[index]), float(y[index + 1])
|
||||
denominator = y0 - 2.0 * y1 + y2
|
||||
if not np.isfinite(denominator) or denominator == 0.0:
|
||||
return float("nan")
|
||||
delta = 0.5 * (y0 - y2) / denominator
|
||||
if not np.isfinite(delta) or abs(delta) > 1.0:
|
||||
return float("nan")
|
||||
step = float(x[index + 1] - x[index])
|
||||
return float(x[index] + delta * step)
|
||||
|
||||
|
||||
def half_max_crossings(x: np.ndarray, net: np.ndarray, index: int,
|
||||
max_span: int) -> tuple[float, float]:
|
||||
"""Linearly interpolated x positions where the net signal falls to half max."""
|
||||
|
||||
peak = float(net[index])
|
||||
if not np.isfinite(peak) or peak <= 0.0:
|
||||
return float("nan"), float("nan")
|
||||
half = 0.5 * peak
|
||||
span = max(1, int(max_span))
|
||||
left = index
|
||||
floor_index = max(0, index - span)
|
||||
while left > floor_index and net[left] > half:
|
||||
left -= 1
|
||||
right = index
|
||||
ceiling = min(net.size - 1, index + span)
|
||||
while right < ceiling and net[right] > half:
|
||||
right += 1
|
||||
|
||||
left_x = right_x = float("nan")
|
||||
if left < index and net[left] <= half:
|
||||
rise = float(net[left + 1] - net[left])
|
||||
left_x = float(x[left] + (half - net[left]) / rise * (x[left + 1] - x[left])) \
|
||||
if rise > 0 else float(x[left])
|
||||
if right > index and net[right] <= half:
|
||||
fall = float(net[right - 1] - net[right])
|
||||
right_x = float(x[right] - (net[right] - half) / fall * (x[right] - x[right - 1])) \
|
||||
if fall > 0 else float(x[right])
|
||||
return left_x, right_x
|
||||
|
||||
|
||||
def _core(x: np.ndarray, amplitude: float, center: float, width: float,
|
||||
shape: str) -> np.ndarray:
|
||||
offset = x - center
|
||||
if shape == "lorentz":
|
||||
return amplitude / (1.0 + (offset / width) ** 2)
|
||||
return amplitude * np.exp(-0.5 * (offset / width) ** 2)
|
||||
|
||||
|
||||
def _shape_model(x: np.ndarray, params: np.ndarray, xref: float, shape: str) -> np.ndarray:
|
||||
amplitude, center, width, c0, c1 = params
|
||||
return _core(x, amplitude, center, width, shape) + c0 + c1 * (x - xref)
|
||||
|
||||
|
||||
def _multi_model(x: np.ndarray, params: np.ndarray, xref: float, shape: str,
|
||||
peaks: int) -> np.ndarray:
|
||||
"""1..N shape components plus a linear background; params = 3*N + 2."""
|
||||
|
||||
total = np.zeros_like(x, dtype=float)
|
||||
for index in range(peaks):
|
||||
amplitude, center, width = params[3 * index:3 * index + 3]
|
||||
total = total + _core(x, amplitude, center, width, shape)
|
||||
return total + params[3 * peaks] + params[3 * peaks + 1] * (x - xref)
|
||||
|
||||
|
||||
def fit_profile(x: np.ndarray, y: np.ndarray, start: list[float], shape: str,
|
||||
bounds: list[tuple[float, float]], max_iter: int = 200,
|
||||
peaks: int = 1, shared_width: bool = False) -> dict:
|
||||
"""Fit 1..N peak components + linear background; returns parameters and flags.
|
||||
|
||||
``shared_width`` fits one common line width for all components, which is the
|
||||
physically sensible and numerically stable model for an unresolved doublet.
|
||||
Parameter layout: per component ``A, mu(, sigma)`` then ``c0, c1``; with a
|
||||
shared width the single ``sigma`` is appended before ``c0, c1``.
|
||||
"""
|
||||
|
||||
x = np.asarray(x, dtype=float)
|
||||
y = np.asarray(y, dtype=float)
|
||||
peaks = int(peaks)
|
||||
per_peak = 2 if shared_width else 3
|
||||
background = per_peak * peaks
|
||||
if x.size < background + 3 or not np.all(np.isfinite(y)):
|
||||
return {"ok": False}
|
||||
xref = float(np.mean(x))
|
||||
lower = np.array([bound[0] for bound in bounds], dtype=float)
|
||||
upper = np.array([bound[1] for bound in bounds], dtype=float)
|
||||
params = np.clip(np.asarray(start, dtype=float), lower, upper)
|
||||
|
||||
def model(values, parameters):
|
||||
shared = parameters[2 * peaks] if shared_width else None
|
||||
total = np.zeros_like(values, dtype=float)
|
||||
for index in range(peaks):
|
||||
if shared_width:
|
||||
amplitude, center = parameters[2 * index], parameters[2 * index + 1]
|
||||
width = shared
|
||||
else:
|
||||
amplitude, center, width = parameters[3 * index:3 * index + 3]
|
||||
total = total + _core(values, amplitude, center, width, shape)
|
||||
return total + parameters[background] + parameters[background + 1] * (values - xref)
|
||||
|
||||
residual = y - model(x, params)
|
||||
cost = float(np.sum(residual * residual))
|
||||
if not np.isfinite(cost):
|
||||
return {"ok": False}
|
||||
lam = 1e-3
|
||||
converged = False
|
||||
for _ in range(max_iter):
|
||||
base = model(x, params)
|
||||
jacobian = np.empty((x.size, params.size))
|
||||
for j in range(params.size):
|
||||
step = 1e-6 * max(1.0, abs(params[j]))
|
||||
probe = params.copy()
|
||||
probe[j] += step
|
||||
jacobian[:, j] = (model(x, probe) - base) / step
|
||||
normal = jacobian.T @ jacobian
|
||||
gradient = jacobian.T @ residual
|
||||
diagonal = np.diag(np.diag(normal)) + 1e-12 * np.eye(params.size)
|
||||
relative = 1.0
|
||||
stepped = False
|
||||
for _ in range(40):
|
||||
try:
|
||||
delta = np.linalg.solve(normal + lam * diagonal, gradient)
|
||||
except np.linalg.LinAlgError:
|
||||
lam *= 10.0
|
||||
continue
|
||||
trial = np.clip(params + delta, lower, upper)
|
||||
trial_residual = y - model(x, trial)
|
||||
trial_cost = float(np.sum(trial_residual * trial_residual))
|
||||
if np.isfinite(trial_cost) and trial_cost < cost:
|
||||
relative = (cost - trial_cost) / max(cost, 1e-30)
|
||||
params, residual, cost = trial, trial_residual, trial_cost
|
||||
lam = max(lam / 3.0, 1e-15)
|
||||
stepped = True
|
||||
break
|
||||
lam *= 10.0
|
||||
if not stepped or relative < 1e-12:
|
||||
converged = True
|
||||
break
|
||||
|
||||
rms = float(np.sqrt(cost / max(1, x.size)))
|
||||
# Parameter errors from the normal matrix at the final parameters; scale the
|
||||
# columns first because amplitude, centre and width differ by orders of magnitude.
|
||||
try:
|
||||
base = model(x, params)
|
||||
jacobian = np.empty((x.size, params.size))
|
||||
for j in range(params.size):
|
||||
step = 1e-6 * max(1.0, abs(params[j]))
|
||||
probe = params.copy()
|
||||
probe[j] += step
|
||||
jacobian[:, j] = (model(x, probe) - base) / step
|
||||
normal = jacobian.T @ jacobian
|
||||
scale = np.sqrt(np.maximum(np.diag(normal), 1e-300))
|
||||
correlation = normal / np.outer(scale, scale)
|
||||
try:
|
||||
inverse = np.linalg.inv(correlation)
|
||||
except np.linalg.LinAlgError:
|
||||
inverse = np.linalg.pinv(correlation)
|
||||
variance = cost / max(1, x.size - params.size)
|
||||
errors = np.sqrt(np.clip(np.diag(inverse) * variance, 0.0, None)) / scale
|
||||
except (np.linalg.LinAlgError, ValueError, FloatingPointError):
|
||||
errors = np.full(params.size, float("nan"))
|
||||
components = []
|
||||
usable = bool(converged and np.isfinite(params).all())
|
||||
for index in range(peaks):
|
||||
if shared_width:
|
||||
amplitude = float(params[2 * index])
|
||||
center = float(params[2 * index + 1])
|
||||
width = float(params[2 * peaks])
|
||||
low, high = lower[2 * peaks], upper[2 * peaks]
|
||||
amplitude_err = float(errors[2 * index])
|
||||
center_err = float(errors[2 * index + 1])
|
||||
width_err = float(errors[2 * peaks])
|
||||
else:
|
||||
amplitude = float(params[3 * index])
|
||||
center = float(params[3 * index + 1])
|
||||
width = float(params[3 * index + 2])
|
||||
low, high = lower[3 * index + 2], upper[3 * index + 2]
|
||||
amplitude_err = float(errors[3 * index])
|
||||
center_err = float(errors[3 * index + 1])
|
||||
width_err = float(errors[3 * index + 2])
|
||||
width_at_bound = bool(width <= low * (1 + 1e-6) or width >= high * (1 - 1e-6))
|
||||
if amplitude <= 0 or width <= 0 or width_at_bound:
|
||||
usable = False
|
||||
if not x[0] - 1.0 <= center <= x[-1] + 1.0:
|
||||
usable = False
|
||||
fwhm = 2.0 * width if shape == "lorentz" else GAUSS_TO_FWHM * width
|
||||
components.append({"amplitude": amplitude, "center": center,
|
||||
"width": width, "fwhm": float(fwhm),
|
||||
"amplitude_err": amplitude_err, "center_err": center_err,
|
||||
"width_err": width_err,
|
||||
"width_at_bound": width_at_bound})
|
||||
first = components[0]
|
||||
return {
|
||||
"ok": usable,
|
||||
"peaks": peaks,
|
||||
"params": params.tolist(),
|
||||
"xref": xref,
|
||||
"components": components,
|
||||
"center": first["center"],
|
||||
"width": first["width"],
|
||||
"amplitude": first["amplitude"],
|
||||
"fwhm": first["fwhm"],
|
||||
"c0": float(params[background]),
|
||||
"c1": float(params[background + 1]),
|
||||
"rms": rms,
|
||||
"converged": converged,
|
||||
"at_bound": any(item["width_at_bound"] for item in components),
|
||||
"n_points": int(x.size),
|
||||
}
|
||||
|
||||
|
||||
def model_values(x, params, xref: float, shape: str) -> np.ndarray:
|
||||
if not params:
|
||||
return np.full(np.shape(x), np.nan, dtype=float)
|
||||
return _shape_model(np.asarray(x, dtype=float), np.asarray(params, dtype=float), xref, shape)
|
||||
|
||||
|
||||
def peak_prominence(net: np.ndarray, index: int) -> tuple[float, float]:
|
||||
"""True topographic prominence of a local maximum and the level it sits on.
|
||||
|
||||
Walk outwards until the profile rises above the peak or the array ends; the
|
||||
prominence base is the higher of the two minima on the way. Unlike a fixed
|
||||
window this keeps resolved doublets apart and rejects shoulders.
|
||||
"""
|
||||
|
||||
height = float(net[index])
|
||||
left = index
|
||||
left_min = height
|
||||
while left > 0:
|
||||
left -= 1
|
||||
value = float(net[left])
|
||||
if value > height:
|
||||
break
|
||||
if value < left_min:
|
||||
left_min = value
|
||||
right = index
|
||||
right_min = height
|
||||
while right < net.size - 1:
|
||||
right += 1
|
||||
value = float(net[right])
|
||||
if value > height:
|
||||
break
|
||||
if value < right_min:
|
||||
right_min = value
|
||||
base = max(left_min, right_min)
|
||||
return height - base, base
|
||||
|
||||
|
||||
def find_peak_candidates(net: np.ndarray, min_distance: int, min_height: float,
|
||||
min_prominence: float, max_peaks: int) -> list[dict]:
|
||||
"""Local maxima with plateau centring, true prominence and neighbour suppression."""
|
||||
|
||||
n = net.size
|
||||
if n < 3 or max_peaks <= 0:
|
||||
return []
|
||||
candidates = []
|
||||
index = 1
|
||||
while index < n - 1:
|
||||
if net[index] > net[index - 1]:
|
||||
end = index
|
||||
while end + 1 < n and net[end + 1] == net[index]:
|
||||
end += 1
|
||||
if end + 1 < n and net[end] > net[end + 1]:
|
||||
candidates.append((index + end) // 2)
|
||||
index = end + 1
|
||||
else:
|
||||
index += 1
|
||||
if not candidates:
|
||||
return []
|
||||
|
||||
candidates.sort(key=lambda i: float(net[i]), reverse=True)
|
||||
kept: list[dict] = []
|
||||
for index in candidates:
|
||||
height = float(net[index])
|
||||
if height < min_height:
|
||||
continue
|
||||
if any(abs(index - other["sample"]) < max(1, int(min_distance)) for other in kept):
|
||||
continue
|
||||
prominence, base = peak_prominence(net, index)
|
||||
if prominence < min_prominence:
|
||||
continue
|
||||
kept.append({"sample": int(index), "height_adu": height,
|
||||
"base_adu": base, "prominence_adu": float(prominence)})
|
||||
if len(kept) >= max_peaks:
|
||||
break
|
||||
return kept
|
||||
|
||||
|
||||
def refine_peak(x: np.ndarray, net: np.ndarray, raw: np.ndarray, candidate: dict,
|
||||
noise: float, step_px: float, fwhm_span: int, fit_window: int,
|
||||
shape: str, saturated: bool, baseline_value: float = 0.0) -> dict:
|
||||
"""Measure one detected peak: centre, width, height, area and fit quality."""
|
||||
|
||||
index = candidate["sample"]
|
||||
n = net.size
|
||||
left_x, right_x = half_max_crossings(x, net, index, fwhm_span)
|
||||
fwhm_measured = (right_x - left_x
|
||||
if np.isfinite(left_x) and np.isfinite(right_x) else float("nan"))
|
||||
|
||||
if fit_window > 0:
|
||||
window_px = float(fit_window) * step_px
|
||||
elif np.isfinite(fwhm_measured):
|
||||
window_px = float(np.clip(2.0 * fwhm_measured, 8.0 * step_px, 90.0 * step_px))
|
||||
else:
|
||||
window_px = 15.0 * step_px
|
||||
lower = int(index)
|
||||
while lower > 0 and x[index] - x[lower] < window_px:
|
||||
lower -= 1
|
||||
upper = int(index)
|
||||
while upper < n - 1 and x[upper] - x[index] < window_px:
|
||||
upper += 1
|
||||
|
||||
center_parabola = parabola_center(x, net, index)
|
||||
|
||||
start_center = center_parabola if np.isfinite(center_parabola) else float(x[index])
|
||||
start_width = fwhm_measured / (2.0 if shape == "lorentz" else GAUSS_TO_FWHM)
|
||||
if not np.isfinite(start_width) or start_width <= 0:
|
||||
start_width = 2.0 * step_px
|
||||
span_px = float(x[upper] - x[lower])
|
||||
bounds = [
|
||||
(0.0, 10.0 * candidate["height_adu"] + 1.0),
|
||||
(x[lower] - span_px, x[upper] + span_px),
|
||||
(0.4 * step_px, max(4.0 * start_width, 0.5 * span_px)),
|
||||
(-5.0 * candidate["height_adu"] - 1.0, 5.0 * candidate["height_adu"] + 1.0),
|
||||
(-1e4, 1e4),
|
||||
]
|
||||
fit = fit_profile(x[lower:upper + 1], net[lower:upper + 1],
|
||||
[candidate["height_adu"], start_center, start_width, 0.0, 0.0],
|
||||
shape, bounds)
|
||||
if fit.get("ok") and (fit["amplitude"] < 0.25 * candidate["height_adu"]
|
||||
or fit["rms"] > 0.2 * fit["amplitude"]):
|
||||
# Reject a fit that misses the measured peak: keep the robust estimators.
|
||||
fit["ok"] = False
|
||||
|
||||
if fit.get("ok"):
|
||||
center, fwhm, source = fit["center"], fit["fwhm"], "fit"
|
||||
elif np.isfinite(center_parabola):
|
||||
# Model-free vertex of the measured maximum: still a peak position.
|
||||
center, fwhm, source = center_parabola, fwhm_measured, "apex"
|
||||
else:
|
||||
# Flat or saturated top makes the parabola unusable; report the sample
|
||||
# where the maximum was detected rather than a window-dependent moment.
|
||||
center, fwhm, source = float(x[index]), fwhm_measured, "sample"
|
||||
|
||||
height = float(net[index])
|
||||
if fit.get("ok") and fit["amplitude"] > height:
|
||||
height = float(fit["amplitude"])
|
||||
return {
|
||||
"sample_index": int(index),
|
||||
"sensor_x_px": float(x[index]),
|
||||
"center_px": float(center),
|
||||
"center_source": source,
|
||||
"center_parabola_px": float(center_parabola) if np.isfinite(center_parabola) else float("nan"),
|
||||
"center_fit_px": float(fit["center"]) if fit.get("ok") else float("nan"),
|
||||
"center_err_px": float(fit["components"][0]["center_err"]) if fit.get("ok") else float("nan"),
|
||||
"sigma_px": float(fit["width"]) if fit.get("ok") else float("nan"),
|
||||
"fwhm_px": float(fwhm) if np.isfinite(fwhm) else float("nan"),
|
||||
"fwhm_halfmax_px": float(fwhm_measured) if np.isfinite(fwhm_measured) else float("nan"),
|
||||
"height_net_adu": height,
|
||||
"prominence_adu": float(candidate["prominence_adu"]),
|
||||
"raw_adu": float(raw[index]),
|
||||
"baseline_adu": float(baseline_value),
|
||||
"snr": float(height / noise) if noise > 0 else float("inf"),
|
||||
"area_net_adu_px": float(np.sum(np.maximum(net[lower:upper + 1], 0.0)) * step_px),
|
||||
"saturated": bool(saturated),
|
||||
"fit_ok": bool(fit.get("ok")),
|
||||
"fit_rms_adu": float(fit["rms"]) if fit.get("ok") else float("nan"),
|
||||
"fit_shape": shape,
|
||||
"fit_params": fit.get("params") if fit.get("ok") else None,
|
||||
"fit_xref": fit.get("xref") if fit.get("ok") else None,
|
||||
"deblended": False,
|
||||
"window_lo_px": float(x[lower]),
|
||||
"window_hi_px": float(x[upper]),
|
||||
"window_lo_sample": int(lower),
|
||||
"window_hi_sample": int(upper),
|
||||
}
|
||||
|
||||
|
||||
def _fit_pair(x: np.ndarray, net: np.ndarray, first: dict, second: dict,
|
||||
shape: str, step_px: float, noise: float) -> Optional[list[dict]]:
|
||||
"""Joint two-component fit of a blended pair; None if it is not trustworthy."""
|
||||
|
||||
lower = min(first["window_lo_sample"], second["window_lo_sample"])
|
||||
upper = max(first["window_hi_sample"], second["window_hi_sample"])
|
||||
xx = x[lower:upper + 1]
|
||||
yy = net[lower:upper + 1]
|
||||
if xx.size < 11:
|
||||
return None
|
||||
starts: list[float] = []
|
||||
bounds: list[tuple[float, float]] = []
|
||||
separation = abs(float(second["sensor_x_px"]) - float(first["sensor_x_px"]))
|
||||
widths = []
|
||||
for peak in (first, second):
|
||||
height = max(float(peak["height_net_adu"]), 1.0)
|
||||
# Start from the detected maximum, not from the biased single-peak fit.
|
||||
center = float(peak["sensor_x_px"])
|
||||
width = float(peak["sigma_px"]) if np.isfinite(peak["sigma_px"]) and peak["sigma_px"] > 0 \
|
||||
else 2.0 * step_px
|
||||
widths.append(width)
|
||||
starts.extend([height, center])
|
||||
bounds.extend([(0.0, 10.0 * height + 1.0),
|
||||
(center - 3.5 * step_px, center + 3.5 * step_px)])
|
||||
# One shared line width: two components of the same blend come from the same
|
||||
# instrument, and independent widths make the fit ill-conditioned.
|
||||
shared = max(0.4 * step_px, min(max(widths), 0.5 * max(separation, step_px)))
|
||||
starts.append(shared)
|
||||
bounds.append((0.4 * step_px, max(8.0 * max(widths), 4.0 * step_px)))
|
||||
scale = max(float(first["height_net_adu"]), float(second["height_net_adu"]), 1.0)
|
||||
starts.extend([0.0, 0.0])
|
||||
bounds.extend([(-3.0 * scale, 3.0 * scale), (-1e4, 1e4)])
|
||||
|
||||
fit = fit_profile(xx, yy, starts, shape, bounds, peaks=2, shared_width=True)
|
||||
if not fit.get("ok"):
|
||||
return None
|
||||
components = fit["components"]
|
||||
if components[1]["center"] - components[0]["center"] < 0.5 * step_px:
|
||||
# The two components collapsed onto each other: this is one feature.
|
||||
return None
|
||||
if abs(components[0]["center"] - first["sensor_x_px"]) > 3.5 * step_px:
|
||||
return None
|
||||
if abs(components[1]["center"] - second["sensor_x_px"]) > 3.5 * step_px:
|
||||
return None
|
||||
|
||||
updated = []
|
||||
for peak, component in zip((first, second), components):
|
||||
merged = dict(peak)
|
||||
merged.update({
|
||||
"center_px": float(component["center"]),
|
||||
"center_fit_px": float(component["center"]),
|
||||
"center_err_px": float(component["center_err"]),
|
||||
"center_source": "pair",
|
||||
"sigma_px": float(component["width"]),
|
||||
"fwhm_px": float(component["fwhm"]),
|
||||
"height_net_adu": float(component["amplitude"]),
|
||||
"snr": float(component["amplitude"] / noise) if noise > 0 else float("inf"),
|
||||
"fit_ok": True,
|
||||
"fit_rms_adu": float(fit["rms"]),
|
||||
"fit_params": [component["amplitude"], component["center"],
|
||||
component["width"], 0.0, 0.0],
|
||||
"fit_xref": float(fit["xref"]),
|
||||
"deblended": True,
|
||||
})
|
||||
updated.append(merged)
|
||||
return updated
|
||||
|
||||
|
||||
def deblend_close_peaks(x: np.ndarray, net: np.ndarray, peaks: list[dict], shape: str,
|
||||
step_px: float, noise: float, blend_factor: float = 1.2) -> list[dict]:
|
||||
"""Refit unresolved pairs jointly instead of pulling their centres together."""
|
||||
|
||||
ordered = sorted(peaks, key=lambda item: item["sensor_x_px"])
|
||||
result: list[dict] = []
|
||||
index = 0
|
||||
while index < len(ordered):
|
||||
current = ordered[index]
|
||||
if index + 1 < len(ordered):
|
||||
neighbour = ordered[index + 1]
|
||||
width_a = current["fwhm_px"] if np.isfinite(current["fwhm_px"]) else 4.0 * step_px
|
||||
width_b = neighbour["fwhm_px"] if np.isfinite(neighbour["fwhm_px"]) else 4.0 * step_px
|
||||
# Use the detected maxima: the fitted centres of a blend can cross.
|
||||
separation = float(neighbour["sensor_x_px"] - current["sensor_x_px"])
|
||||
if 0.0 < separation <= blend_factor * 0.5 * (width_a + width_b):
|
||||
pair = _fit_pair(x, net, current, neighbour, shape, step_px, noise)
|
||||
if pair is not None:
|
||||
result.extend(pair)
|
||||
index += 2
|
||||
continue
|
||||
result.append(current)
|
||||
index += 1
|
||||
return result
|
||||
|
||||
|
||||
def dedupe_channel_peaks(peaks: list[dict], min_distance_px: float) -> list[dict]:
|
||||
"""Drop repeated maxima closer than the peak distance, using detection positions.
|
||||
|
||||
The fitted centre of a blended line is pulled towards its neighbour, so the
|
||||
distance is measured between the raw local maxima (``sensor_x_px``) and not
|
||||
between the refined centres; otherwise a resolved doublet can collapse.
|
||||
"""
|
||||
|
||||
kept: list[dict] = []
|
||||
for peak in sorted(peaks, key=lambda item: item["height_net_adu"], reverse=True):
|
||||
if any(abs(peak["sensor_x_px"] - other["sensor_x_px"]) < min_distance_px
|
||||
for other in kept):
|
||||
continue
|
||||
kept.append(peak)
|
||||
kept.sort(key=lambda item: item["sensor_x_px"])
|
||||
return kept
|
||||
|
||||
|
||||
def summarise_line(members: dict[str, dict]) -> dict:
|
||||
"""Combine the per-channel peaks of one line into a line record (no id)."""
|
||||
|
||||
for channel, item in members.items():
|
||||
item["channel"] = channel
|
||||
items = list(members.values())
|
||||
strongest = max((item["height_net_adu"] for item in items), default=0.0)
|
||||
weights = np.array([
|
||||
max(0.0, item["height_net_adu"]) if item["height_net_adu"] >= 0.25 * strongest else 0.0
|
||||
for item in items
|
||||
], dtype=float)
|
||||
positions = np.array([item["center_px"] for item in items], dtype=float)
|
||||
center = (float(np.dot(weights, positions) / weights.sum())
|
||||
if weights.sum() > 0 else float(np.mean(positions)))
|
||||
primary = max(items, key=lambda item: item["snr"])
|
||||
return {
|
||||
"sensor_x_px": center,
|
||||
"center_spread_px": float(positions.max() - positions.min()),
|
||||
"channels": members,
|
||||
"channel_names": "".join(c for c in ("R", "G", "B") if c in members),
|
||||
"n_channels": len(members),
|
||||
"primary_channel": primary["channel"],
|
||||
"height_net_adu": primary["height_net_adu"],
|
||||
"max_height_net_adu": float(strongest),
|
||||
"snr": primary["snr"],
|
||||
"fwhm_px": primary["fwhm_px"],
|
||||
"area_net_adu_px": primary["area_net_adu_px"],
|
||||
"saturated": any(item["saturated"] for item in items),
|
||||
"sample_index": primary["sample_index"],
|
||||
"sample_step_px": None,
|
||||
}
|
||||
|
||||
|
||||
def merge_coincident_lines(lines: list[dict], tolerance_px: float = 1.5) -> list[dict]:
|
||||
"""Join lines that ended up at the same place in complementary channels.
|
||||
|
||||
Weak lines can be detected a few pixels apart in different CFA channels; if
|
||||
the refined centres agree to better than one sample they are one line.
|
||||
"""
|
||||
|
||||
merged: list[dict] = []
|
||||
for line in sorted(lines, key=lambda item: item["sensor_x_px"]):
|
||||
target = None
|
||||
for other in merged:
|
||||
if abs(line["sensor_x_px"] - other["sensor_x_px"]) < tolerance_px:
|
||||
target = other
|
||||
break
|
||||
if target is None:
|
||||
merged.append({"sensor_x_px": line["sensor_x_px"],
|
||||
"channels": dict(line["channels"])})
|
||||
continue
|
||||
combined = dict(target["channels"])
|
||||
for channel, peak in line["channels"].items():
|
||||
other = combined.get(channel)
|
||||
if other is None or peak["snr"] > other["snr"]:
|
||||
combined[channel] = peak
|
||||
target["channels"] = combined
|
||||
|
||||
summaries = [summarise_line(item["channels"]) for item in merged]
|
||||
summaries.sort(key=lambda entry: entry["sensor_x_px"])
|
||||
for index, summary in enumerate(summaries, 1):
|
||||
summary["id"] = f"P{index:03d}"
|
||||
return summaries
|
||||
|
||||
|
||||
def group_channels(peaks: dict[str, list[dict]], tolerance_px: float,
|
||||
merge_tolerance_px: float = 1.5) -> list[dict]:
|
||||
"""Merge per-channel peaks that belong to the same spectral line."""
|
||||
|
||||
# Cluster on the detected local maxima, not on the fitted centres: a single
|
||||
# peak fit inside a blend is pulled towards its neighbour and would wrongly
|
||||
# merge two resolved lines.
|
||||
flat = [(item["sensor_x_px"], channel, item)
|
||||
for channel, items in peaks.items() for item in items]
|
||||
flat.sort(key=lambda row: row[0])
|
||||
groups: list[list[tuple[float, str, dict]]] = []
|
||||
for entry in flat:
|
||||
if groups and entry[0] - groups[-1][-1][0] <= tolerance_px:
|
||||
groups[-1].append(entry)
|
||||
else:
|
||||
groups.append([entry])
|
||||
|
||||
lines = []
|
||||
for group in groups:
|
||||
members: dict[str, dict] = {}
|
||||
for _, channel, item in group:
|
||||
other = members.get(channel)
|
||||
# One line cannot have two centres in the same CFA channel.
|
||||
if other is None or item["snr"] > other["snr"]:
|
||||
members[channel] = item
|
||||
summary = summarise_line(dict(members))
|
||||
summary["id"] = ""
|
||||
lines.append(summary)
|
||||
return merge_coincident_lines(lines, merge_tolerance_px)
|
||||
|
||||
|
||||
def attach_wavelength(lines: list[dict], calibration: Optional[Calibration]) -> None:
|
||||
"""Add wavelength, extrapolation flag and per-channel wavelength data."""
|
||||
|
||||
for line in lines:
|
||||
for item in line["channels"].values():
|
||||
if calibration is not None:
|
||||
item["wavelength_nm"] = float(calibration(item["center_px"]))
|
||||
item["extrapolated"] = bool(
|
||||
item["center_px"] < calibration.span[0] or item["center_px"] > calibration.span[1])
|
||||
item["dispersion_nm_per_px"] = float(calibration.dispersion(item["center_px"]))
|
||||
item["fwhm_nm"] = float(abs(item["fwhm_px"] * item["dispersion_nm_per_px"]))
|
||||
else:
|
||||
item["wavelength_nm"] = float("nan")
|
||||
item["extrapolated"] = False
|
||||
item["dispersion_nm_per_px"] = float("nan")
|
||||
item["fwhm_nm"] = float("nan")
|
||||
if calibration is not None:
|
||||
line["wavelength_nm"] = float(calibration(line["sensor_x_px"]))
|
||||
line["extrapolated"] = bool(
|
||||
line["sensor_x_px"] < calibration.span[0] or line["sensor_x_px"] > calibration.span[1])
|
||||
line["dispersion_nm_per_px"] = float(calibration.dispersion(line["sensor_x_px"]))
|
||||
line["fwhm_nm"] = float(abs(line["fwhm_px"] * line["dispersion_nm_per_px"]))
|
||||
else:
|
||||
line["wavelength_nm"] = float("nan")
|
||||
line["extrapolated"] = False
|
||||
line["dispersion_nm_per_px"] = float("nan")
|
||||
line["fwhm_nm"] = float("nan")
|
||||
line.update({"match_species": "", "match_nm": float("nan"),
|
||||
"match_delta_nm": float("nan"), "match_label": ""})
|
||||
line["x_plot"] = line["wavelength_nm"] if calibration is not None else line["sensor_x_px"]
|
||||
@@ -0,0 +1,551 @@
|
||||
import csv
|
||||
import json
|
||||
import os
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
|
||||
import numpy as np
|
||||
|
||||
import peak_finder as pf
|
||||
import spectro_core as core
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
TEST_TMP = os.environ.get("PEAK_FINDER_TEST_TMP")
|
||||
|
||||
|
||||
def _find_recording(name: str) -> Path:
|
||||
"""Look for recordings/ here and one or two levels up.
|
||||
|
||||
The recordings often live beside the program folder (Light/recordings vs
|
||||
Light/Programs), so a fixed path would silently skip the end-to-end tests.
|
||||
"""
|
||||
|
||||
for base in (HERE, HERE.parent, HERE.parent.parent):
|
||||
candidate = base / "recordings" / name
|
||||
if candidate.is_dir():
|
||||
return candidate
|
||||
return HERE / "recordings" / name
|
||||
|
||||
|
||||
# Optional overrides let the end-to-end tests run against a recording kept elsewhere.
|
||||
RECORDING = Path(os.environ.get("PEAK_FINDER_TEST_RECORDING",
|
||||
_find_recording("hq_20261101_114417")))
|
||||
CALIBRATION = Path(os.environ.get("PEAK_FINDER_TEST_CALIBRATION",
|
||||
HERE / "wavelength_calibration" / "calibration.json"))
|
||||
MODULE_DIR = Path(os.environ.get("PEAK_FINDER_TEST_MODULE_DIR", HERE))
|
||||
AVAILABLE = RECORDING.is_dir() and CALIBRATION.is_file()
|
||||
|
||||
|
||||
def read_rows(path) -> list[dict]:
|
||||
with open(path, newline="", encoding="utf-8") as handle:
|
||||
return list(csv.DictReader(handle))
|
||||
|
||||
|
||||
class temp_directory:
|
||||
"""Scratch directory context manager; PEAK_FINDER_TEST_TMP overrides the location.
|
||||
|
||||
A plain ``mkdir`` is used instead of ``tempfile.mkdtemp`` so the directory
|
||||
keeps the parent's permissions, which matters inside restrictive sandboxes.
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
import shutil
|
||||
import uuid
|
||||
|
||||
self._shutil = shutil
|
||||
base = Path(TEST_TMP) if TEST_TMP else Path(tempfile.gettempdir())
|
||||
base.mkdir(parents=True, exist_ok=True)
|
||||
self.path = base / f"pf_case_{uuid.uuid4().hex[:10]}"
|
||||
self.path.mkdir()
|
||||
|
||||
def __enter__(self) -> str:
|
||||
return str(self.path)
|
||||
|
||||
def __exit__(self, *exc_info) -> bool:
|
||||
self._shutil.rmtree(self.path, ignore_errors=True)
|
||||
return False
|
||||
|
||||
|
||||
def gaussian(x, amplitude, center, sigma, base=0.0, slope=0.0, xref=None):
|
||||
xref = float(np.mean(x)) if xref is None else xref
|
||||
return base + slope * (x - xref) + amplitude * np.exp(-0.5 * ((x - center) / sigma) ** 2)
|
||||
|
||||
|
||||
def make_peak(center, height, fwhm, snr, channel="G", **extra):
|
||||
peak = {
|
||||
"channel": channel,
|
||||
"sample_index": int(round(center / 2.0)),
|
||||
"sensor_x_px": float(round(center / 2.0) * 2.0),
|
||||
"center_px": float(center),
|
||||
"center_source": "fit",
|
||||
"center_parabola_px": float(center) + 0.3,
|
||||
"center_fit_px": float(center),
|
||||
"sigma_px": fwhm / 2.3548200450309493,
|
||||
"fwhm_px": fwhm,
|
||||
"fwhm_halfmax_px": fwhm * 1.02,
|
||||
"height_net_adu": height,
|
||||
"prominence_adu": height * 0.9,
|
||||
"raw_adu": height + 100.0,
|
||||
"baseline_adu": 10.0,
|
||||
"snr": snr,
|
||||
"area_net_adu_px": height * fwhm,
|
||||
"saturated": False,
|
||||
"fit_ok": True,
|
||||
"fit_rms_adu": height * 0.02,
|
||||
"fit_shape": "gauss",
|
||||
"fit_params": [height, float(center), fwhm / 2.3548200450309493, 10.0, 0.0],
|
||||
"fit_xref": float(center),
|
||||
"window_lo_px": float(center) - 20.0,
|
||||
"window_hi_px": float(center) + 20.0,
|
||||
"window_lo_sample": 0,
|
||||
"window_hi_sample": 10,
|
||||
}
|
||||
peak.update(extra)
|
||||
return peak
|
||||
|
||||
|
||||
def make_lines():
|
||||
wavelengths = [404.7, 435.8, 546.1, 577.0, 579.1]
|
||||
return [
|
||||
{"id": f"P{index:03d}", "wavelength_nm": value, "sensor_x_px": 200.0 + 100.0 * index,
|
||||
"snr": float(10 * index), "center_spread_px": 0.5, "extrapolated": False}
|
||||
for index, value in enumerate(wavelengths, 1)
|
||||
]
|
||||
|
||||
|
||||
class EstimatorTests(unittest.TestCase):
|
||||
def test_baseline_follows_continuum_not_peaks(self):
|
||||
x = np.arange(0, 600, 2.0)
|
||||
y = 100.0 + 0.05 * x + gaussian(x, 500.0, 300.0, 5.0)
|
||||
baseline = pf.estimate_baseline(y, 50, lambda values, width: values)
|
||||
error = np.abs(baseline - (100.0 + 0.05 * x))[60:-60]
|
||||
self.assertLess(float(np.max(error)), 5.0)
|
||||
|
||||
def test_baseline_disabled_returns_zero(self):
|
||||
y = np.arange(10.0)
|
||||
np.testing.assert_array_equal(pf.estimate_baseline(y, 0, lambda v, w: v), np.zeros(10))
|
||||
|
||||
def test_noise_proxy_of_white_noise(self):
|
||||
rng = np.random.default_rng(7)
|
||||
y = rng.normal(0.0, 7.0, 4000)
|
||||
self.assertAlmostEqual(pf.estimate_noise(y), 7.0, delta=0.5)
|
||||
|
||||
def test_parabola_and_half_max_width(self):
|
||||
x = np.arange(0, 200, 2.0)
|
||||
y = gaussian(x, 1000.0, 101.0, 4.0)
|
||||
index = int(np.argmax(y))
|
||||
self.assertAlmostEqual(pf.parabola_center(x, y, index), 101.0, delta=0.5)
|
||||
left, right = pf.half_max_crossings(x, y, index, 40)
|
||||
self.assertAlmostEqual(right - left, 2.3548200450309493 * 4.0, delta=0.6)
|
||||
|
||||
|
||||
class FitTests(unittest.TestCase):
|
||||
def test_gaussian_fit_recovers_parameters(self):
|
||||
rng = np.random.default_rng(11)
|
||||
x = np.arange(600.0, 700.0, 2.0)
|
||||
truth = gaussian(x, 1200.0, 651.7, 3.4, base=40.0, slope=0.8, xref=650.0)
|
||||
y = truth + rng.normal(0.0, 8.0, x.size)
|
||||
bounds = [(0.0, 20000.0), (580.0, 720.0), (0.8, 40.0), (-2000.0, 2000.0), (-100.0, 100.0)]
|
||||
fit = pf.fit_profile(x, y, [1000.0, 650.0, 4.0, 0.0, 0.0], "gauss", bounds)
|
||||
self.assertTrue(fit["ok"])
|
||||
self.assertAlmostEqual(fit["center"], 651.7, delta=0.4)
|
||||
self.assertAlmostEqual(fit["fwhm"], 2.3548200450309493 * 3.4, delta=0.6)
|
||||
self.assertAlmostEqual(fit["amplitude"], 1200.0, delta=120.0)
|
||||
|
||||
def test_lorentzian_fit_recovers_width(self):
|
||||
rng = np.random.default_rng(3)
|
||||
x = np.arange(0.0, 120.0, 2.0)
|
||||
y = 900.0 / (1.0 + ((x - 61.2) / 3.0) ** 2) + rng.normal(0.0, 5.0, x.size)
|
||||
bounds = [(0.0, 10000.0), (20.0, 100.0), (1.0, 20.0), (-500.0, 500.0), (-50.0, 50.0)]
|
||||
fit = pf.fit_profile(x, y, [900.0, 60.0, 3.0, 0.0, 0.0], "lorentz", bounds)
|
||||
self.assertTrue(fit["ok"])
|
||||
self.assertAlmostEqual(fit["fwhm"], 6.0, delta=0.6)
|
||||
|
||||
def test_model_values_round_trip(self):
|
||||
x = np.arange(0.0, 10.0)
|
||||
params = [5.0, 4.0, 1.5, 1.0, 0.0]
|
||||
model = pf.model_values(x, params, 4.0, "gauss")
|
||||
self.assertAlmostEqual(float(np.max(model)), 6.0, delta=1e-6)
|
||||
self.assertTrue(np.all(np.isfinite(model)))
|
||||
|
||||
|
||||
class DetectionTests(unittest.TestCase):
|
||||
def test_three_peaks_found_in_continuum(self):
|
||||
rng = np.random.default_rng(5)
|
||||
x = np.arange(0.0, 800.0, 2.0)
|
||||
y = (0.02 * x + gaussian(x, 1000.0, 200.0, 4.0) + gaussian(x, 400.0, 400.0, 6.0)
|
||||
+ gaussian(x, 120.0, 600.0, 3.0) + rng.normal(0.0, 2.0, x.size))
|
||||
baseline = pf.estimate_baseline(y, 50, lambda values, width: values)
|
||||
net = y - baseline
|
||||
noise = pf.estimate_noise(net)
|
||||
candidates = pf.find_peak_candidates(net, 4, 20.0, 20.0, 10)
|
||||
centers = sorted(x[item["sample"]] for item in candidates)
|
||||
self.assertEqual(len(centers), 3)
|
||||
for found, expected in zip(centers, (200.0, 400.0, 600.0)):
|
||||
self.assertAlmostEqual(found, expected, delta=4.0)
|
||||
self.assertLess(noise, 5.0)
|
||||
|
||||
def test_refined_centres_are_sub_sample(self):
|
||||
rng = np.random.default_rng(9)
|
||||
x = np.arange(0.0, 200.0, 2.0)
|
||||
y = gaussian(x, 800.0, 97.3, 2.6) + rng.normal(0.0, 4.0, x.size)
|
||||
baseline = pf.estimate_baseline(y, 40, lambda values, width: values)
|
||||
net = y - baseline
|
||||
noise = pf.estimate_noise(net)
|
||||
candidate = pf.find_peak_candidates(net, 4, 20.0, 20.0, 4)[0]
|
||||
peak = pf.refine_peak(x, net, y, candidate, noise, 2.0, 30, 0, "gauss", False)
|
||||
self.assertAlmostEqual(peak["center_px"], 97.3, delta=0.6)
|
||||
self.assertAlmostEqual(peak["fwhm_px"], 2.3548200450309493 * 2.6, delta=0.8)
|
||||
self.assertTrue(peak["fit_ok"])
|
||||
|
||||
def test_silent_profile_has_no_candidates(self):
|
||||
net = np.zeros(50)
|
||||
self.assertEqual(pf.find_peak_candidates(net, 4, 1.0, 1.0, 5), [])
|
||||
|
||||
|
||||
class BlendTests(unittest.TestCase):
|
||||
def test_failed_fit_falls_back_to_the_measured_maximum(self):
|
||||
from unittest.mock import patch
|
||||
|
||||
x = np.arange(400.0, 700.0, 2.0)
|
||||
net = (gaussian(x, 10000.0, 542.0, 15.0)
|
||||
+ 4000.0 * np.exp(-np.clip(x - 542.0, 0.0, None) / 120.0))
|
||||
candidate = {"sample": int(np.argmax(net)), "height_adu": float(net.max()),
|
||||
"base_adu": 0.0, "prominence_adu": float(net.max())}
|
||||
with patch.object(core, "fit_profile", return_value={"ok": False}):
|
||||
peak = pf.refine_peak(x, net, net, candidate, 1.0, 2.0, 30, 0, "gauss", False)
|
||||
self.assertEqual(peak["center_source"], "apex")
|
||||
self.assertAlmostEqual(peak["center_px"], 542.0, delta=1.0)
|
||||
|
||||
def test_flat_top_falls_back_to_the_sampled_maximum(self):
|
||||
from unittest.mock import patch
|
||||
|
||||
x = np.arange(0.0, 60.0, 2.0)
|
||||
net = np.full(x.size, 100.0)
|
||||
net[10:16] = 1000.0 # saturated flat top
|
||||
candidate = {"sample": 12, "height_adu": 1000.0,
|
||||
"base_adu": 100.0, "prominence_adu": 900.0}
|
||||
with patch.object(core, "fit_profile", return_value={"ok": False}):
|
||||
peak = pf.refine_peak(x, net, net, candidate, 1.0, 2.0, 30, 0, "gauss", False)
|
||||
self.assertEqual(peak["center_source"], "sample")
|
||||
self.assertAlmostEqual(peak["center_px"], 24.0) # x[12]
|
||||
|
||||
def test_topographic_prominence_rejects_a_shoulder(self):
|
||||
net = np.array([0.0, 100.0, 200.0, 500.0, 300.0, 400.0, 200.0, 0.0])
|
||||
prominence, base = pf.peak_prominence(net, 5)
|
||||
self.assertAlmostEqual(prominence, 100.0)
|
||||
self.assertAlmostEqual(base, 300.0)
|
||||
prominence, _ = pf.peak_prominence(net, 3)
|
||||
self.assertAlmostEqual(prominence, 500.0)
|
||||
|
||||
def test_resolved_doublet_survives_detection(self):
|
||||
rng = np.random.default_rng(21)
|
||||
x = np.arange(780.0, 830.0, 2.0)
|
||||
net = (gaussian(x, 3600.0, 800.0, 2.5) + gaussian(x, 4300.0, 808.0, 2.5)
|
||||
+ rng.normal(0.0, 20.0, x.size))
|
||||
candidates = pf.find_peak_candidates(net, 2, 200.0, 100.0, 10)
|
||||
self.assertEqual(len(candidates), 2)
|
||||
self.assertEqual(sorted(round(float(x[item["sample"]]), 1) for item in candidates),
|
||||
[800.0, 808.0])
|
||||
|
||||
def test_doublet_is_deblended_with_one_shared_width(self):
|
||||
rng = np.random.default_rng(22)
|
||||
x = np.arange(780.0, 830.0, 2.0)
|
||||
net = (gaussian(x, 3600.0, 800.0, 2.5) + gaussian(x, 4300.0, 808.0, 2.5)
|
||||
+ rng.normal(0.0, 20.0, x.size))
|
||||
noise = pf.estimate_noise(net)
|
||||
candidates = pf.find_peak_candidates(net, 2, 200.0, 100.0, 10)
|
||||
peaks = [pf.refine_peak(x, net, net, item, noise, 2.0, 30, 0, "gauss", False)
|
||||
for item in candidates]
|
||||
deduped = pf.dedupe_channel_peaks(peaks, 4.0)
|
||||
self.assertEqual(len(deduped), 2)
|
||||
merged = pf.deblend_close_peaks(x, net, deduped, "gauss", 2.0, noise)
|
||||
self.assertEqual(len(merged), 2)
|
||||
self.assertTrue(all(peak["deblended"] for peak in merged))
|
||||
centres = sorted(peak["center_px"] for peak in merged)
|
||||
self.assertAlmostEqual(centres[0], 800.0, delta=0.8)
|
||||
self.assertAlmostEqual(centres[1], 808.0, delta=0.8)
|
||||
self.assertEqual(len({round(peak["fwhm_px"], 6) for peak in merged}), 1)
|
||||
self.assertAlmostEqual(merged[0]["fwhm_px"], 2.3548200450309493 * 2.5, delta=0.8)
|
||||
for peak in merged:
|
||||
self.assertTrue(np.isfinite(peak["center_err_px"]))
|
||||
|
||||
def test_coincident_weak_lines_are_merged(self):
|
||||
first = pf.summarise_line({"B": make_peak(912.0, 430.0, 9.2, 75.0, "B")})
|
||||
second = pf.summarise_line({"R": make_peak(917.0, 70.0, 8.0, 15.0, "R")})
|
||||
first["id"] = second["id"] = ""
|
||||
lines = [dict(first, sensor_x_px=912.0), dict(second, sensor_x_px=917.0)]
|
||||
self.assertEqual(len(pf.merge_coincident_lines(lines, 1.5)), 2)
|
||||
lines[1]["sensor_x_px"] = 912.4
|
||||
combined = pf.merge_coincident_lines(lines, 1.5)
|
||||
self.assertEqual(len(combined), 1)
|
||||
self.assertEqual(combined[0]["channel_names"], "RB")
|
||||
|
||||
|
||||
class GroupingTests(unittest.TestCase):
|
||||
def test_same_line_in_three_channels_merges(self):
|
||||
peaks = {
|
||||
"R": [make_peak(500.0, 300.0, 9.0, 40.0, "R")],
|
||||
"G": [make_peak(501.0, 1200.0, 10.0, 200.0, "G")],
|
||||
"B": [make_peak(499.5, 800.0, 11.0, 90.0, "B")],
|
||||
}
|
||||
lines = pf.group_channels(peaks, 4.0)
|
||||
self.assertEqual(len(lines), 1)
|
||||
line = lines[0]
|
||||
self.assertEqual(line["channel_names"], "RGB")
|
||||
self.assertEqual(line["primary_channel"], "G")
|
||||
self.assertEqual(line["id"], "P001")
|
||||
self.assertAlmostEqual(line["sensor_x_px"], (300 * 500 + 1200 * 501 + 800 * 499.5) / 2300,
|
||||
places=6)
|
||||
self.assertAlmostEqual(line["center_spread_px"], 1.5, places=6)
|
||||
|
||||
def test_distinct_lines_stay_separate(self):
|
||||
peaks = {"G": [make_peak(500.0, 100.0, 8.0, 20.0), make_peak(520.0, 90.0, 8.0, 18.0)]}
|
||||
lines = pf.group_channels(peaks, 4.0)
|
||||
self.assertEqual([line["id"] for line in lines], ["P001", "P002"])
|
||||
|
||||
def test_channel_duplicate_is_dropped(self):
|
||||
peaks = {"G": [make_peak(500.0, 300.0, 9.0, 50.0), make_peak(500.4, 900.0, 9.0, 90.0)]}
|
||||
lines = pf.group_channels(peaks, 4.0)
|
||||
self.assertEqual(len(lines), 1)
|
||||
self.assertEqual(lines[0]["height_net_adu"], 900.0)
|
||||
|
||||
def test_dedupe_keeps_neighbours_further_than_distance(self):
|
||||
peaks = [make_peak(100.0, 500.0, 8.0, 40.0), make_peak(112.0, 400.0, 8.0, 30.0),
|
||||
make_peak(103.0, 300.0, 8.0, 20.0)]
|
||||
kept = pf.dedupe_channel_peaks(peaks, 12.0)
|
||||
self.assertEqual([peak["center_px"] for peak in kept], [100.0, 112.0])
|
||||
|
||||
|
||||
class OutputNamingTests(unittest.TestCase):
|
||||
def test_default_output_names_the_frame(self):
|
||||
single = pf.default_output_directory(r"C:\raw\hq_20261112_200710", [7])
|
||||
self.assertEqual(single.name, "hq_20261112_200710_f0007")
|
||||
stacked = pf.default_output_directory(r"C:\raw\hq_20261112_200710", [5, 6, 7])
|
||||
self.assertEqual(stacked.name, "hq_20261112_200710_f0005-0007_avg3")
|
||||
self.assertEqual(stacked.parent.name, "peak_exports")
|
||||
|
||||
|
||||
class SelectionTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.lines = make_lines()
|
||||
|
||||
def test_all_and_none(self):
|
||||
self.assertEqual(len(pf.parse_selection("all", self.lines)), 5)
|
||||
self.assertEqual(pf.parse_selection("none", self.lines), set())
|
||||
self.assertEqual(pf.parse_selection("ALL", self.lines), pf.parse_selection("*", self.lines))
|
||||
|
||||
def test_ids_and_ranges(self):
|
||||
self.assertEqual(pf.parse_selection("2", self.lines), {"P002"})
|
||||
self.assertEqual(pf.parse_selection("1-2", self.lines), {"P001", "P002"})
|
||||
self.assertEqual(pf.parse_selection("id:4", self.lines), {"P004"})
|
||||
|
||||
def test_wavelength_ranges(self):
|
||||
self.assertEqual(pf.parse_selection("nm:540-560", self.lines), {"P003"})
|
||||
self.assertEqual(pf.parse_selection("540-560", self.lines), {"P003"})
|
||||
self.assertEqual(pf.parse_selection("546.0", self.lines), {"P003"})
|
||||
self.assertEqual(pf.parse_selection("nm:430-440", self.lines), {"P002"})
|
||||
|
||||
def test_top_n_and_union(self):
|
||||
self.assertEqual(pf.parse_selection("top:2", self.lines), {"P004", "P005"})
|
||||
self.assertEqual(pf.parse_selection("1,nm:540-560", self.lines), {"P001", "P003"})
|
||||
|
||||
def test_bad_tokens_raise(self):
|
||||
for token in ("9", "0", "abc", "id:3-9", "nm:"):
|
||||
with self.assertRaises(ValueError):
|
||||
pf.parse_selection(token, self.lines)
|
||||
|
||||
def test_wavelength_selection_needs_calibration(self):
|
||||
lines = [dict(line, wavelength_nm=float("nan")) for line in self.lines]
|
||||
with self.assertRaises(ValueError):
|
||||
pf.parse_selection("nm:500-600", lines)
|
||||
self.assertEqual(pf.parse_selection("2", lines), {"P002"})
|
||||
|
||||
|
||||
class CalibrationTests(unittest.TestCase):
|
||||
def write(self, payload):
|
||||
handle = tempfile.NamedTemporaryFile("w", suffix=".json", delete=False, encoding="utf-8", dir=TEST_TMP)
|
||||
handle.write(json.dumps(payload))
|
||||
handle.close()
|
||||
return handle.name
|
||||
|
||||
def test_scale_offset_and_inverse(self):
|
||||
path = self.write({"coefficients_descending": [0.285, 349.45],
|
||||
"calibrated_pixel_range": [304.0, 808.5]})
|
||||
calibration = pf.Calibration(path, offset_nm=0.5)
|
||||
calibration.set_width(2028)
|
||||
self.assertAlmostEqual(float(calibration(600.0)), 0.285 * 600 + 349.45 + 0.5, places=6)
|
||||
self.assertAlmostEqual(float(calibration.dispersion(600.0)), 0.285, places=9)
|
||||
self.assertAlmostEqual(float(calibration.inverse(calibration(600.0))), 600.0, delta=1.0)
|
||||
self.assertAlmostEqual(calibration.nm_span[0], 0.285 * 304 + 349.95, places=6)
|
||||
|
||||
def test_invalid_calibrations(self):
|
||||
for payload in ({"coefficients_descending": [0.0, 1.0], "calibrated_pixel_range": [1, 2]},
|
||||
{"coefficients_descending": [1.0], "calibrated_pixel_range": [1, 2]},
|
||||
{"coefficients_descending": [1.0, 2.0], "calibrated_pixel_range": [3, 2]},
|
||||
{"coefficients_descending": [1.0, 2.0]}):
|
||||
with self.assertRaises(ValueError):
|
||||
pf.Calibration(self.write(payload))
|
||||
|
||||
def test_geometry_checks(self):
|
||||
path = self.write({"coefficients_descending": [0.285, 349.45],
|
||||
"calibrated_pixel_range": [304.0, 808.5]})
|
||||
calibration = pf.Calibration(path)
|
||||
source = SimpleNamespace(full_size=(2028, 1520), pattern="BGGR")
|
||||
calibration.validate_geometry(source, lambda *a, **k: None)
|
||||
bad = pf.Calibration(self.write({"coefficients_descending": [0.001, -2.0, 900.0],
|
||||
"calibrated_pixel_range": [304.0, 808.5]}))
|
||||
with self.assertRaises(ValueError):
|
||||
bad.validate_geometry(source, lambda *a, **k: None)
|
||||
|
||||
|
||||
class LineTableTests(unittest.TestCase):
|
||||
def test_match_nearest_within_tolerance(self):
|
||||
table = [{"wavelength_nm": 546.0, "species": "Hg I", "label": "546"},
|
||||
{"wavelength_nm": 579.1, "species": "Hg I", "label": "579"}]
|
||||
lines = [{"id": "P001", "wavelength_nm": 546.4, "sensor_x_px": 1.0},
|
||||
{"id": "P002", "wavelength_nm": 600.0, "sensor_x_px": 2.0}]
|
||||
pf.match_lines(lines, table, 1.5)
|
||||
self.assertEqual(lines[0]["match_species"], "Hg I")
|
||||
self.assertAlmostEqual(lines[0]["match_delta_nm"], 0.4, places=6)
|
||||
self.assertEqual(lines[1]["match_species"], "")
|
||||
self.assertTrue(np.isnan(lines[1]["match_delta_nm"]))
|
||||
|
||||
def test_csv_table_with_flexible_columns(self):
|
||||
with temp_directory() as folder:
|
||||
path = Path(folder) / "lines.csv"
|
||||
path.write_text("nm,element\n546.0735,Hg I\n435.8328,Hg I\n", encoding="utf-8")
|
||||
table = pf.load_line_table(path)
|
||||
self.assertEqual([entry["wavelength_nm"] for entry in table], [435.8328, 546.0735])
|
||||
|
||||
|
||||
def build_spectrum_and_lines():
|
||||
"""Run the pipeline up to detection, for tests that need real peaks."""
|
||||
|
||||
hq = pf.load_hq(str(MODULE_DIR))
|
||||
args = pf.build_parser().parse_args(["--input", str(RECORDING), "--output", "unused",
|
||||
"--quiet"])
|
||||
source = hq.RecordingSource(str(RECORDING))
|
||||
source.start()
|
||||
image, original, indices, frame_meta = pf.read_frames(hq, source, args, lambda *a, **k: None)
|
||||
roi = hq.detect_auto_roi(hq.make_preview(original, source.pattern), *source.full_size)
|
||||
source.close()
|
||||
meta = dict(source.base_meta)
|
||||
meta.update({key: value for key, value in frame_meta.items()
|
||||
if not str(key).startswith("_")})
|
||||
black, full = hq.resolve_levels(meta, source.bit_depth, args.black, args.full_scale)
|
||||
spectrum = pf.SpectrumData(hq, source, image, original, indices, roi, args, black, full)
|
||||
lines = pf.group_channels(spectrum.detect(), args.group_tol)
|
||||
pf.attach_wavelength(lines, None)
|
||||
return spectrum, lines
|
||||
|
||||
|
||||
@unittest.skipUnless(AVAILABLE, "recordings or calibration not available")
|
||||
class PickerTests(unittest.TestCase):
|
||||
def test_hover_previews_and_click_selects(self):
|
||||
import matplotlib
|
||||
matplotlib.use("Agg")
|
||||
import matplotlib.pyplot as plt
|
||||
from matplotlib.backend_bases import KeyEvent, MouseEvent
|
||||
|
||||
spectrum, lines = build_spectrum_and_lines()
|
||||
self.assertGreaterEqual(len(lines), 3)
|
||||
target = lines[2]
|
||||
seen = {}
|
||||
|
||||
def fake_show(block=True):
|
||||
figure = plt.gcf()
|
||||
canvas = figure.canvas
|
||||
axes = figure.axes[0]
|
||||
x_display = float(axes.transData.transform((target["x_plot"], 0.0))[0])
|
||||
y_display = float(axes.transAxes.transform((0.0, 0.5))[1])
|
||||
canvas.callbacks.process(
|
||||
"motion_notify_event", MouseEvent("motion_notify_event", canvas,
|
||||
x_display, y_display))
|
||||
seen["axes"] = len(figure.axes)
|
||||
seen["images"] = len(figure.axes[1].images)
|
||||
seen["curves"] = len(figure.axes[2].lines)
|
||||
seen["title"] = figure.axes[2].get_title()
|
||||
canvas.callbacks.process(
|
||||
"button_press_event", MouseEvent("button_press_event", canvas,
|
||||
x_display, y_display, button=1))
|
||||
canvas.callbacks.process(
|
||||
"key_press_event", KeyEvent("key_press_event", canvas, key="e"))
|
||||
|
||||
real_show = plt.show
|
||||
plt.show = fake_show
|
||||
try:
|
||||
chosen = pf.pick_peaks(spectrum, lines, None, set())
|
||||
finally:
|
||||
plt.show = real_show
|
||||
self.assertEqual(chosen, {target["id"]})
|
||||
self.assertEqual(seen["axes"], 3) # spectrum + RAW patch + zoom
|
||||
self.assertGreaterEqual(seen["images"], 1) # RAW patch rendered
|
||||
self.assertGreaterEqual(seen["curves"], 3) # R/G/B profiles drawn
|
||||
self.assertIn(target["id"], seen["title"])
|
||||
|
||||
|
||||
@unittest.skipUnless(AVAILABLE, "recordings or calibration not available")
|
||||
class EndToEndTests(unittest.TestCase):
|
||||
def test_export_selected_peaks(self):
|
||||
with temp_directory() as folder:
|
||||
code = pf.main(["--input", str(RECORDING), "--calibration", str(CALIBRATION),
|
||||
"--output", folder, "--select", "nm:530-560",
|
||||
"--no-images", "--quiet", "--module-dir", str(MODULE_DIR)])
|
||||
self.assertEqual(code, 0)
|
||||
output = Path(folder)
|
||||
rows = read_rows(output / "selected_peaks.csv")
|
||||
self.assertTrue(rows)
|
||||
for row in rows:
|
||||
self.assertGreaterEqual(float(row["wavelength_nm"]), 530.0)
|
||||
self.assertLessEqual(float(row["wavelength_nm"]), 560.0)
|
||||
self.assertTrue(0 <= float(row["patch_x0"]) < float(row["patch_x1"]) <= 2028)
|
||||
self.assertTrue(0 <= float(row["patch_y0"]) < float(row["patch_y1"]) <= 1520)
|
||||
self.assertTrue(0 <= float(row["sensor_x_px"]))
|
||||
self.assertTrue((output / row["shape_csv"]).is_file())
|
||||
manifest = json.loads((output / "peaks.json").read_text(encoding="utf-8"))
|
||||
all_rows = read_rows(output / "peaks_all.csv")
|
||||
self.assertGreaterEqual(len(all_rows), len(manifest["peaks"]))
|
||||
self.assertEqual(manifest["calibration"]["wavelength_offset_nm"], 0.0)
|
||||
self.assertTrue(manifest["coordinate_convention"]["sensor_x_px"])
|
||||
|
||||
def test_without_calibration_only_pixels_are_exported(self):
|
||||
with temp_directory() as folder:
|
||||
code = pf.main(["--input", str(RECORDING), "--output", folder, "--select", "top:1",
|
||||
"--no-images", "--quiet", "--module-dir", str(MODULE_DIR)])
|
||||
self.assertEqual(code, 0)
|
||||
row = read_rows(Path(folder) / "selected_peaks.csv")[0]
|
||||
self.assertEqual(row["wavelength_nm"], "")
|
||||
self.assertGreater(float(row["sensor_x_px"]), 0.0)
|
||||
manifest = json.loads((Path(folder) / "peaks.json").read_text(encoding="utf-8"))
|
||||
self.assertIsNone(manifest["calibration"])
|
||||
|
||||
def test_bad_frame_is_rejected(self):
|
||||
code = pf.main(["--input", str(RECORDING), "--frame", "100000", "--quiet",
|
||||
"--output", str(Path(tempfile.gettempdir()) / "pf_unused"),
|
||||
"--no-images", "--module-dir", str(MODULE_DIR)])
|
||||
self.assertEqual(code, 2)
|
||||
|
||||
def test_wavelength_offset_shifts_the_scale(self):
|
||||
with temp_directory() as folder:
|
||||
code = pf.main(["--input", str(RECORDING), "--calibration", str(CALIBRATION),
|
||||
"--wavelength-offset", "-10", "--output", folder,
|
||||
"--select", "top:1", "--no-images", "--quiet",
|
||||
"--module-dir", str(MODULE_DIR)])
|
||||
self.assertEqual(code, 0)
|
||||
row = read_rows(Path(folder) / "selected_peaks.csv")[0]
|
||||
self.assertAlmostEqual(float(row["wavelength_nm"]), 545.8237 - 10.0, delta=0.2)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
import json
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
import numpy as np
|
||||
import wavelength_player as wp
|
||||
|
||||
|
||||
def calibration(coeff=(.6, -8), span=(690, 927)):
|
||||
data = dict(coefficients_descending=list(coeff), calibrated_pixel_range=list(span),
|
||||
roi=[620, 620, 1034, 848])
|
||||
with patch.object(Path, 'read_text', return_value=json.dumps(data)):
|
||||
return wp.SavedCalibration('dummy.json')
|
||||
|
||||
|
||||
class PlayerTests(unittest.TestCase):
|
||||
def test_exact_saved_coefficients(self):
|
||||
model = calibration()
|
||||
np.testing.assert_array_equal(model.coeff, [.6, -8])
|
||||
np.testing.assert_allclose(model([690, 742, 926.5]), [406, 437.2, 547.9])
|
||||
|
||||
def test_invalid_models(self):
|
||||
for coeff, span in (([0, 3], [1, 2]), ([float('nan'), 1], [1, 2]),
|
||||
([1], [1, 2]), ([1, 2], [4, 3])):
|
||||
with self.assertRaises(ValueError):
|
||||
calibration(coeff, span)
|
||||
|
||||
def test_quadratic_and_reverse_geometry(self):
|
||||
source = SimpleNamespace(full_size=(2028, 1520), pattern='BGGR')
|
||||
calibration([.00001, .5, 5]).validate_geometry(source)
|
||||
calibration([-.5, 1000]).validate_geometry(source)
|
||||
with self.assertRaises(ValueError):
|
||||
calibration([.001, -1, 500]).validate_geometry(source)
|
||||
|
||||
def test_geometry_mismatch(self):
|
||||
model = calibration()
|
||||
model.data['recording'] = 'reference'
|
||||
with patch.object(Path, 'is_file', return_value=True), patch.object(Path, 'read_text',
|
||||
return_value=json.dumps(dict(width=4056, height=3040))):
|
||||
with self.assertRaises(ValueError):
|
||||
model.validate_geometry(SimpleNamespace(full_size=(2028, 1520), pattern='BGGR'))
|
||||
|
||||
def test_processing_signed_signal_and_sensor_coordinates(self):
|
||||
hq = wp.load_player()
|
||||
model = calibration()
|
||||
wp.install_hooks(hq, model)
|
||||
args = hq.parse_args(['--smooth', '1', '--black', '10'])
|
||||
raw = np.full((16, 64), 8, dtype=np.uint16)
|
||||
raw[:, 30:32] = 110
|
||||
roi = hq.make_roi(10, 0, 60, 16, 64, 16)
|
||||
result = hq.process_frame(raw, {}, roi, 'BGGR', args, model, (64, 16), 'SBGGR12', 12)
|
||||
for c in hq.CHANNELS:
|
||||
self.assertEqual(result.signal_raw[c][0], -2)
|
||||
np.testing.assert_array_equal(result.signal[c], result.signal_raw[c])
|
||||
np.testing.assert_allclose(result.x_plot[c], model(result.x_px[c]))
|
||||
self.assertEqual(result.normalized[c].max(), 1)
|
||||
self.assertGreaterEqual(result.x_px[c][0], 10)
|
||||
|
||||
def test_help(self):
|
||||
self.assertEqual(wp.main(['--help']), 0)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -0,0 +1,276 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Independent Hg wavelength calibration; uses hq_spectrum recording/CFA APIs.
|
||||
|
||||
Peak finding uses the same core as peak_finder.py: topographic prominence,
|
||||
Gaussian centre with a peak-top fallback, and a joint two-line fit for blended
|
||||
doublets (the Hg 576.96/579.07 pair is only ~6 px apart). Baseline subtraction
|
||||
is used for detection and peak heights only; `signal_adu` still reports the
|
||||
black-corrected value at the peak sample.
|
||||
|
||||
Sensor coordinates are full-resolution columns.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import argparse
|
||||
import csv
|
||||
import json
|
||||
from pathlib import Path
|
||||
import sys
|
||||
import numpy as np
|
||||
import hq_spectrum as hq
|
||||
try:
|
||||
import spectro_core as core
|
||||
except ImportError as exc: # pragma: no cover - placement error
|
||||
raise SystemExit('wavelength_calibration.py needs spectro_core.py in the same directory '
|
||||
'(or on sys.path)') from exc
|
||||
|
||||
HG = (404.6563, 435.8328, 546.0735, 576.9598, 579.0663)
|
||||
SOURCE = 'https://physics.nist.gov/PhysRefData/Handbook/Tables/mercurytable2.htm'
|
||||
|
||||
|
||||
def parser():
|
||||
p = argparse.ArgumentParser(description=__doc__)
|
||||
p.add_argument('--input', required=True, help='hq_record recording directory')
|
||||
p.add_argument('--frame', type=int, default=0, help='zero-based frame index (default 0)')
|
||||
p.add_argument('--roi', default='auto', help='auto or x0,y0,x1,y1; same as hq_spectrum')
|
||||
p.add_argument('--black', type=float, help='black level in decoded ADU')
|
||||
p.add_argument('--full-scale', type=float)
|
||||
p.add_argument('--reduce', choices=['mean', 'median'], default='mean')
|
||||
p.add_argument('--baseline-window', type=float, default=100,
|
||||
help='baseline opening width in sensor px; 0 disables the baseline')
|
||||
p.add_argument('--peak-distance', type=float, default=4,
|
||||
help='minimum distance in sensor pixels per channel (2 px = one sample)')
|
||||
p.add_argument('--peak-prominence', type=float, default=0.01,
|
||||
help='topographic prominence as a fraction of the channel maximum')
|
||||
p.add_argument('--peak-height', type=float, default=0.02,
|
||||
help='peak height as a fraction of the channel maximum')
|
||||
p.add_argument('--min-snr', type=float, default=15, help='peak height in units of the noise')
|
||||
p.add_argument('--max-peaks', type=int, default=40, help='per channel')
|
||||
p.add_argument('--fwhm-span', type=float, default=60, help='half-width search range in px')
|
||||
p.add_argument('--fit-window', type=float, default=0, help='fit window half width in px; 0 auto')
|
||||
p.add_argument('--shape', choices=['gauss', 'lorentz'], default='gauss')
|
||||
p.add_argument('--assign', help='confirmed candidate identities, e.g. B1:404.6563,B2:435.8328,G3:546.0735')
|
||||
p.add_argument('--order', type=int, choices=[1, 2], default=1)
|
||||
p.add_argument('--headless', action='store_true', help='save candidates; fit only with explicit --assign')
|
||||
p.add_argument('--output', default='wavelength_calibration', help='output directory')
|
||||
return p
|
||||
|
||||
|
||||
def extract(raw, meta, roi, pattern, depth, args):
|
||||
planes = hq.extract_planes(raw, pattern, roi)
|
||||
counts = {c: planes[c].shape[1] for c in hq.CHANNELS}
|
||||
xs = hq.channel_x_positions(roi, pattern, counts['G'], min(counts['R'], counts['B']))
|
||||
black, full = hq.resolve_levels(meta, depth, args.black, args.full_scale)
|
||||
window = 0 if args.baseline_window <= 0 else max(3, int(round(args.baseline_window / 2.0)))
|
||||
curves, candidates = {}, []
|
||||
for c in hq.CHANNELS:
|
||||
y = (np.median(planes[c], axis=0) if args.reduce == 'median'
|
||||
else np.mean(planes[c], axis=0, dtype=np.float64)) - black
|
||||
x = xs[c][:len(y)]
|
||||
baseline = core.estimate_baseline(y, window, hq.smooth_vector)
|
||||
net = y - baseline
|
||||
noise = core.estimate_noise(net)
|
||||
top = float(np.max(net)) if net.size else 0.0
|
||||
positive = np.maximum(net, 0.0)
|
||||
norm = positive / positive.max() if positive.max() > 0 else np.zeros(net.size)
|
||||
peaks = []
|
||||
if top > 0 and norm.size:
|
||||
min_height = max(args.peak_height * top, args.min_snr * noise)
|
||||
min_prominence = max(args.peak_prominence * top, 0.4 * args.min_snr * noise)
|
||||
min_distance = max(1, int(round(args.peak_distance / 2.0)))
|
||||
found = core.find_peak_candidates(net, min_distance, min_height,
|
||||
min_prominence, args.max_peaks)
|
||||
for candidate in found:
|
||||
index = candidate['sample']
|
||||
peaks.append(core.refine_peak(
|
||||
x, net, y, candidate, noise, 2.0,
|
||||
max(1, int(round(args.fwhm_span / 2.0))),
|
||||
max(0, int(round(args.fit_window / 2.0))),
|
||||
args.shape, core.is_saturated(raw, roi, x[index], full),
|
||||
baseline_value=float(baseline[index])))
|
||||
# Two lines closer than ~1.2 FWHM are fitted jointly, otherwise a
|
||||
# single-peak fit collapses them onto one centre.
|
||||
peaks = core.deblend_close_peaks(
|
||||
x, net, core.dedupe_channel_peaks(peaks, args.peak_distance),
|
||||
args.shape, 2.0, noise)
|
||||
peaks.sort(key=lambda item: item['center_px'])
|
||||
curves[c] = (x, y, baseline, net, norm)
|
||||
for j, peak in enumerate(peaks, 1):
|
||||
index = peak['sample_index']
|
||||
candidates.append(dict(
|
||||
id=f'{c}{j}', channel=c, sample=int(index),
|
||||
pixel=float(peak['center_px']),
|
||||
signal_adu=float(y[index]),
|
||||
height_net_adu=float(peak['height_net_adu']),
|
||||
normalized=float(norm[index]),
|
||||
fwhm_px=float(peak['fwhm_px']),
|
||||
snr=float(peak['snr']),
|
||||
center_source=peak['center_source'],
|
||||
deblended=bool(peak.get('deblended')),
|
||||
saturated=bool(peak['saturated'])))
|
||||
return curves, candidates, black, full
|
||||
|
||||
|
||||
def assignments(text, candidates, order):
|
||||
lookup = {v['id']: v for v in candidates}
|
||||
result, used = [], set()
|
||||
for token in text.split(','):
|
||||
name, wavelength = token.strip().split(':')
|
||||
name = name.upper().strip()
|
||||
if name not in lookup or name in used:
|
||||
raise ValueError(f'unknown or duplicate candidate: {name}')
|
||||
nm = float(wavelength)
|
||||
if not np.isfinite(nm) or nm <= 0:
|
||||
raise ValueError('wavelength must be finite and positive')
|
||||
used.add(name)
|
||||
result.append({**lookup[name], 'wavelength_nm': nm})
|
||||
if len(result) < order + 1:
|
||||
raise ValueError(f'order {order} needs at least {order + 1} confirmed lines')
|
||||
x = np.array([r['pixel'] for r in result])
|
||||
y = np.array([r['wavelength_nm'] for r in result])
|
||||
if len(np.unique(x)) != len(x) or len(np.unique(y)) != len(y):
|
||||
raise ValueError('select one channel candidate per distinct spectral line')
|
||||
sort = np.argsort(x)
|
||||
if not (np.all(np.diff(y[sort]) > 0) or np.all(np.diff(y[sort]) < 0)):
|
||||
raise ValueError('confirmed wavelength order must be monotonic in sensor x')
|
||||
return result
|
||||
|
||||
|
||||
def fit(confirmed, order, width):
|
||||
x = np.array([r['pixel'] for r in confirmed])
|
||||
y = np.array([r['wavelength_nm'] for r in confirmed])
|
||||
model = np.polynomial.Polynomial.fit(x, y, order).convert()
|
||||
coeff = model.coef[::-1]
|
||||
derivative = np.polyval(np.polyder(coeff), [0, width - 1])
|
||||
if np.any(np.abs(derivative) < 1e-12) or derivative[0] * derivative[1] <= 0:
|
||||
raise ValueError('mapping is not monotonic across sensor; revise identities/order/ROI')
|
||||
residual = np.polyval(coeff, x) - y
|
||||
return coeff, residual
|
||||
|
||||
|
||||
def write_csv(path, fields, rows):
|
||||
with path.open('w', newline='', encoding='utf-8') as f:
|
||||
w = csv.writer(f)
|
||||
w.writerow(fields)
|
||||
w.writerows(rows)
|
||||
|
||||
|
||||
def run(args):
|
||||
if args.frame < 0 or args.peak_distance <= 0 or args.max_peaks < 1:
|
||||
raise ValueError('frame >= 0, peak-distance > 0 and max-peaks >= 1 required')
|
||||
for v in (args.peak_height, args.peak_prominence):
|
||||
if not np.isfinite(v) or not 0 <= v <= 1:
|
||||
raise ValueError('normalized peak thresholds must be between 0 and 1')
|
||||
import matplotlib
|
||||
if args.headless:
|
||||
matplotlib.use('Agg')
|
||||
import matplotlib.pyplot as plt
|
||||
out = Path(args.output)
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
source = hq.RecordingSource(args.input)
|
||||
try:
|
||||
source.start()
|
||||
raw, meta = source.read_at(args.frame)
|
||||
roi = hq.parse_roi(args.roi, *source.full_size)
|
||||
preview = hq.make_preview(raw, source.pattern)
|
||||
if roi is None:
|
||||
roi = hq.detect_auto_roi(preview, *source.full_size)
|
||||
if roi is None:
|
||||
raise ValueError('auto ROI failed; specify --roi x0,y0,x1,y1')
|
||||
curves, candidates, black, full = extract(raw, meta, roi, source.pattern,
|
||||
source.bit_depth, args)
|
||||
finally:
|
||||
source.close()
|
||||
fields = ['id', 'channel', 'sample', 'pixel', 'signal_adu', 'height_net_adu',
|
||||
'normalized', 'fwhm_px', 'snr', 'center_source', 'deblended', 'saturated']
|
||||
write_csv(out / 'candidates.csv', fields, ([v[k] for k in fields] for v in candidates))
|
||||
write_csv(out / 'profiles.csv',
|
||||
['channel', 'pixel', 'black_corrected_adu', 'baseline_adu', 'net_adu',
|
||||
'normalized_for_detection'],
|
||||
([c, x, y, b, n, q] for c, arrays in curves.items()
|
||||
for x, y, b, n, q in zip(*arrays)))
|
||||
fig, axes = plt.subplots(3, 1, figsize=(13, 10), constrained_layout=True)
|
||||
axes[0].imshow(preview, extent=(0, source.full_size[0], source.full_size[1], 0))
|
||||
from matplotlib.patches import Rectangle
|
||||
axes[0].add_patch(Rectangle((roi.x0, roi.y0), roi.width, roi.height, fill=False, edgecolor='orange'))
|
||||
axes[0].set_title(f'Frame {args.frame} (zero based); ROI {roi.text()}; black={black:g} ADU')
|
||||
for c, (x, y, baseline, net, norm) in curves.items():
|
||||
axes[1].plot(x, y, color=hq.CHANNEL_COLORS[c], label=c)
|
||||
axes[1].plot(x, baseline, color=hq.CHANNEL_COLORS[c], lw=0.6, ls=':', alpha=0.7)
|
||||
axes[2].plot(x, norm, color=hq.CHANNEL_COLORS[c], label=c)
|
||||
for v in candidates:
|
||||
marker = 's' if v['deblended'] else 'o'
|
||||
axes[2].plot(v['pixel'], v['normalized'], marker, color=hq.CHANNEL_COLORS[v['channel']],
|
||||
mfc='none' if v['deblended'] else hq.CHANNEL_COLORS[v['channel']])
|
||||
axes[2].annotate(v['id'], (v['pixel'], v['normalized']), xytext=(0, 8),
|
||||
textcoords='offset points')
|
||||
print(f"{v['id']:5s} x={v['pixel']:9.3f} FWHM={v['fwhm_px']:7.2f} px "
|
||||
f"SNR={v['snr']:8.0f} normalized={v['normalized']:.4f}"
|
||||
f"{' [pair]' if v['deblended'] else ''}{' [sat]' if v['saturated'] else ''}")
|
||||
axes[1].set_ylabel('Black-corrected ADU'); axes[1].legend()
|
||||
axes[2].set_ylabel('Normalized (detection only)'); axes[2].set_xlabel('Sensor column / px')
|
||||
axes[2].set_ylim(-0.05, 1.18)
|
||||
fig.savefig(out / 'candidates.png', dpi=160)
|
||||
print('Hg I air wavelengths (nm):', HG)
|
||||
print('Blended pairs such as 576.960/579.066 nm (about 7 px apart) are split by a joint '
|
||||
'two-line fit and marked [pair]; use one channel per line.')
|
||||
print('Select ONE channel candidate per line. Review candidates.png before confirming identities.')
|
||||
text = args.assign
|
||||
if not args.headless:
|
||||
print('Inspect/zoom the plot, then CLOSE its window to enter confirmed identities in the terminal.')
|
||||
plt.show(block=True)
|
||||
if text is None:
|
||||
text = input('Confirmed IDs:wavelength (comma separated), blank to export candidates only: ').strip()
|
||||
plt.close(fig)
|
||||
if not text:
|
||||
print(f'Candidates exported to {out.resolve()}; no calibration fitted.')
|
||||
return 0
|
||||
confirmed = assignments(text, candidates, args.order)
|
||||
coeff, residual = fit(confirmed, args.order, source.full_size[0])
|
||||
x = np.array([r['pixel'] for r in confirmed])
|
||||
lo, hi = float(x.min()), float(x.max())
|
||||
info = dict(recording=str(Path(args.input).resolve()), frame_zero_based=args.frame,
|
||||
roi=[roi.x0, roi.y0, roi.x1, roi.y1], pattern=source.pattern,
|
||||
raw_format=source.raw_format, black_adu=black, full_scale_adu=full,
|
||||
reduce=args.reduce, order=args.order, coefficients_descending=coeff.tolist(),
|
||||
formula='wavelength_nm = numpy.polyval(coefficients_descending, sensor_x_px)',
|
||||
wavelength_medium='air', reference=SOURCE, confirmed=confirmed,
|
||||
residual_definition='fitted - reference (nm)', residuals_nm=residual.tolist(),
|
||||
rms_nm=float(np.sqrt(np.mean(residual ** 2))), max_abs_nm=float(np.max(np.abs(residual))),
|
||||
residual_degrees_of_freedom=len(x)-args.order-1, calibrated_pixel_range=[lo, hi],
|
||||
processing='spatial reduction, black subtraction, morphological baseline for '
|
||||
'peak detection, topographic prominence, Gaussian centres with '
|
||||
'peak-top fallback and joint fits for blended doublets')
|
||||
(out / 'calibration.json').write_text(json.dumps(info, indent=2), encoding='utf-8')
|
||||
write_csv(out / 'residuals.csv', ['id', 'pixel', 'reference_nm', 'fitted_nm', 'residual_nm'],
|
||||
([r['id'], r['pixel'], r['wavelength_nm'], r['wavelength_nm']+e, e] for r, e in zip(confirmed, residual)))
|
||||
grid = np.arange(source.full_size[0])
|
||||
write_csv(out / 'pixel_wavelength.csv', ['sensor_x_px', 'wavelength_nm', 'extrapolated'],
|
||||
([int(p), float(w), int(p < lo or p > hi)] for p, w in zip(grid, np.polyval(coeff, grid))))
|
||||
fig, ax = plt.subplots(2, 1, figsize=(10, 7), constrained_layout=True)
|
||||
ax[0].plot(grid, np.polyval(coeff, grid)); ax[0].scatter(x, [r['wavelength_nm'] for r in confirmed])
|
||||
ax[0].axvspan(lo, hi, alpha=.1, color='green', label='Confirmed span'); ax[0].legend()
|
||||
ax[0].set_ylabel('Wavelength / nm (air)')
|
||||
ax[1].axhline(0, color='gray'); ax[1].scatter(x, residual)
|
||||
for r, e in zip(confirmed, residual):
|
||||
ax[1].annotate(r['id'], (r['pixel'], e))
|
||||
ax[1].set_ylabel('Fit - reference / nm'); ax[1].set_xlabel('Sensor column / px')
|
||||
fig.savefig(out / 'calibration.png', dpi=160); plt.close(fig)
|
||||
print('Coefficients (descending):', coeff)
|
||||
print(f"RMS={info['rms_nm']:.6g} nm; max |residual|={info['max_abs_nm']:.6g} nm")
|
||||
if len(x) == args.order + 1:
|
||||
print('Exact interpolation: zero residual does not establish calibration accuracy; add independent lines.')
|
||||
print(f'Calibration exported to {out.resolve()}')
|
||||
return 0
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
args = parser().parse_args(argv)
|
||||
try:
|
||||
return run(args)
|
||||
except (ValueError, RuntimeError, OSError, IndexError, StopIteration, EOFError) as e:
|
||||
print(f'Calibration error: {e or "frame outside recording"}', file=sys.stderr)
|
||||
return 2
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,66 @@
|
||||
{
|
||||
"recording": "D:\\Dev\\Light\\Programs\\recordings\\hq_20261101_114417",
|
||||
"frame_zero_based": 0,
|
||||
"roi": [
|
||||
98,
|
||||
654,
|
||||
848,
|
||||
856
|
||||
],
|
||||
"pattern": "BGGR",
|
||||
"raw_format": "BGGR_PISP_COMP1",
|
||||
"black_adu": 4096.0,
|
||||
"full_scale_adu": 65535.0,
|
||||
"reduce": "mean",
|
||||
"order": 1,
|
||||
"coefficients_descending": [
|
||||
0.2849422617422453,
|
||||
349.4538237085669
|
||||
],
|
||||
"formula": "wavelength_nm = numpy.polyval(coefficients_descending, sensor_x_px)",
|
||||
"wavelength_medium": "air",
|
||||
"reference": "https://physics.nist.gov/PhysRefData/Handbook/Tables/mercurytable2.htm",
|
||||
"confirmed": [
|
||||
{
|
||||
"id": "B1",
|
||||
"channel": "B",
|
||||
"sample": 103,
|
||||
"pixel": 304.0,
|
||||
"signal_adu": 9736.237623762376,
|
||||
"normalized": 1.0,
|
||||
"wavelength_nm": 435.8328
|
||||
},
|
||||
{
|
||||
"id": "G1",
|
||||
"channel": "G",
|
||||
"sample": 294,
|
||||
"pixel": 686.5,
|
||||
"signal_adu": 54219.9801980198,
|
||||
"normalized": 1.0,
|
||||
"wavelength_nm": 546.0735
|
||||
},
|
||||
{
|
||||
"id": "G2",
|
||||
"channel": "G",
|
||||
"sample": 355,
|
||||
"pixel": 808.5,
|
||||
"signal_adu": 17836.9900990099,
|
||||
"normalized": 0.3289744856760633,
|
||||
"wavelength_nm": 579.0663
|
||||
}
|
||||
],
|
||||
"residual_definition": "fitted - reference (nm)",
|
||||
"residuals_nm": [
|
||||
0.24347127820948344,
|
||||
-1.0068136053816943,
|
||||
0.7633423271722677
|
||||
],
|
||||
"rms_nm": 0.7428870276938522,
|
||||
"max_abs_nm": 1.0068136053816943,
|
||||
"residual_degrees_of_freedom": 1,
|
||||
"calibrated_pixel_range": [
|
||||
304.0,
|
||||
808.5
|
||||
],
|
||||
"processing": "spatial reduction, black subtraction; no smoothing; max normalization only for detection"
|
||||
}
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 70 KiB |
@@ -0,0 +1,9 @@
|
||||
id,channel,sample,pixel,signal_adu,normalized
|
||||
R1,R,36,301.0,1066.666666666667,0.2308984047019312
|
||||
R2,R,89,407.0,1468.606060606061,0.31790512174643165
|
||||
R3,R,285,799.0,4619.636363636364,1.0
|
||||
R4,R,345,919.0,1852.121212121212,0.4009235936188077
|
||||
G1,G,285,798.5,61439.0,1.0
|
||||
B1,B,35,298.0,8451.878787878788,0.1652635570724308
|
||||
B2,B,89,406.0,51141.818181818184,1.0
|
||||
B3,B,285,798.0,9526.30303030303,0.1862722791050436
|
||||
|
Binary file not shown.
|
After Width: | Height: | Size: 820 KiB |
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,4 @@
|
||||
id,pixel,reference_nm,fitted_nm,residual_nm
|
||||
B1,304.0,435.8328,436.0762712782095,0.24347127820948344
|
||||
G1,686.5,546.0735,545.0666863946183,-1.0068136053816943
|
||||
G2,808.5,579.0663,579.8296423271722,0.7633423271722677
|
||||
|
@@ -0,0 +1,137 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Replay hq_record RAW recordings with saved wavelength calibration.
|
||||
|
||||
Keep next to hq_spectrum.py and spectro_core.py. All original playback switches
|
||||
are supported.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import argparse
|
||||
from contextlib import redirect_stdout
|
||||
import io
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
import numpy as np
|
||||
|
||||
try: # shared core lives beside this script
|
||||
import spectro_core as core
|
||||
except ImportError as exc: # pragma: no cover - placement error
|
||||
raise SystemExit('wavelength_player.py needs spectro_core.py in the same directory '
|
||||
'(or on sys.path)') from exc
|
||||
|
||||
# Kept as the historical name: the player validates a saved calibration exactly
|
||||
# like every other tool now does.
|
||||
SavedCalibration = core.Calibration
|
||||
|
||||
|
||||
def load_player():
|
||||
# A private module instance provides hooks without editing the original file.
|
||||
return core.load_hq()
|
||||
|
||||
|
||||
def install_hooks(hq, calibration):
|
||||
original_process = hq.process_frame
|
||||
|
||||
def process(raw, meta, roi, pattern, args, model, full_size, raw_format, bit_depth):
|
||||
result = original_process(raw, meta, roi, pattern, args, model, full_size, raw_format, bit_depth)
|
||||
# Retain signed black-subtracted ADU; normalized copies are detection only.
|
||||
planes = hq.extract_planes(raw, pattern, roi)
|
||||
for c in hq.CHANNELS:
|
||||
values = (np.median(planes[c], axis=0) if args.reduce == 'median'
|
||||
else np.mean(planes[c], axis=0, dtype=np.float64)) - result.black
|
||||
values = values[:len(result.x_px[c])]
|
||||
positive = np.maximum(values, 0)
|
||||
norm = positive / positive.max() if positive.size and positive.max() > 0 else np.zeros_like(values)
|
||||
result.signal_raw[c] = values
|
||||
result.signal[c] = values.copy()
|
||||
result.normalized[c] = norm
|
||||
result.peaks[c] = hq.find_peaks_normalized(norm, result.x_px[c],
|
||||
max(1, int(np.ceil(args.peak_distance / 2))), args.peak_prominence,
|
||||
args.peak_height, args.max_peaks)
|
||||
return result
|
||||
|
||||
class CalibratedDashboard(hq.Dashboard):
|
||||
def __init__(self, *args, **kwargs):
|
||||
super().__init__(*args, **kwargs)
|
||||
self.ax_norm.set_title('Normalized profiles (detection only; no smoothing)', fontsize=9)
|
||||
for axis in (self.ax_spec, self.ax_norm, self.ax_water):
|
||||
axis.axvspan(*calibration.nm_span, alpha=.08, color='green', zorder=0)
|
||||
|
||||
def update(self, result, waterfall, fps, status_text):
|
||||
# imshow needs an equally spaced nm grid, including quadratic/reverse mappings.
|
||||
if waterfall.size:
|
||||
gx = result.x_plot['G']
|
||||
xmin = min(float(x.min()) for x in result.x_plot.values())
|
||||
xmax = max(float(x.max()) for x in result.x_plot.values())
|
||||
grid = np.linspace(xmin, xmax, waterfall.shape[1])
|
||||
order = np.argsort(gx)
|
||||
waterfall = np.asarray([np.interp(grid, gx[order], row[order]) for row in waterfall])
|
||||
super().update(result, waterfall, fps, status_text)
|
||||
|
||||
def _target_y_limits(self, result):
|
||||
# Signed signal (black already subtracted) can dip below zero, so the
|
||||
# range is padded; rounding and hysteresis happen in Dashboard, which
|
||||
# keeps the cached background valid instead of redrawing every frame.
|
||||
valid = [v for v in result.signal.values() if v.size]
|
||||
ymin = min(float(v.min()) for v in valid) if valid else 0.0
|
||||
ymax = max(float(v.max()) for v in valid) if valid else 1.0
|
||||
pad = max(1.0, (ymax - ymin) * 0.05)
|
||||
return (hq._nice_floor(min(0.0, ymin - pad)),
|
||||
hq._nice_ceiling(max(1.0, ymax + pad)))
|
||||
|
||||
hq.process_frame = process
|
||||
hq.Dashboard = CalibratedDashboard
|
||||
hq.make_calibration = lambda args: calibration
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
argv = sys.argv[1:] if argv is None else list(argv)
|
||||
options = argparse.ArgumentParser(add_help=False, allow_abbrev=False)
|
||||
options.add_argument('--calibration', default=str(Path(__file__).with_name('wavelength_calibration') / 'calibration.json'))
|
||||
extra, rest = options.parse_known_args(argv)
|
||||
try:
|
||||
hq = load_player()
|
||||
if '-h' in rest or '--help' in rest:
|
||||
print('Wavelength recording player: wavelength_player.py --input REC_DIR --calibration calibration.json')
|
||||
print('--calibration: saved JSON; --roi calibration: use saved ROI; default ROI is auto.')
|
||||
print('No smoothing. Original player options follow:\n')
|
||||
buffer = io.StringIO()
|
||||
with redirect_stdout(buffer):
|
||||
try:
|
||||
hq.parse_args(['--help'])
|
||||
except SystemExit:
|
||||
pass
|
||||
print(buffer.getvalue().replace('(default: 3)', '(default: 1)').replace(
|
||||
'Live R/G/B spectral analysis of Raspberry Pi HQ Camera RAW frames.',
|
||||
'Saved-calibration playback of HQ Camera RAW recordings.'))
|
||||
print('Only recording playback is supported; --demo, --list-modes, --calibrate are unavailable.')
|
||||
return 0
|
||||
args = hq.parse_args(['--smooth', '1'] + rest)
|
||||
if not args.input or args.demo or args.list_modes or args.calibrate:
|
||||
raise ValueError('use --input REC_DIR and --calibration JSON; live/demo/--calibrate are unsupported')
|
||||
if args.smooth != 1:
|
||||
raise ValueError('this player preserves unsmoothed signal; --smooth must be 1')
|
||||
calibration = SavedCalibration(extra.calibration)
|
||||
source = hq.RecordingSource(args.input)
|
||||
calibration.validate_geometry(source)
|
||||
args.width, args.height = source.full_size
|
||||
if args.roi == 'calibration':
|
||||
roi = calibration.data['roi']
|
||||
if len(roi) != 4 or any(not np.isfinite(float(v)) for v in roi):
|
||||
raise ValueError('invalid saved ROI')
|
||||
args.roi = ','.join(str(v) for v in roi)
|
||||
print('Calibration:', calibration.path)
|
||||
print('Descending coefficients:', calibration.coeff.tolist())
|
||||
print(calibration.describe())
|
||||
print('Use unchanged optics and sensor mode; each recording supplies its own black level.')
|
||||
if args.headless and not args.frames:
|
||||
print('Headless mode: processing all frames; use --frames N for a shorter run.')
|
||||
install_hooks(hq, calibration)
|
||||
return hq.run(args)
|
||||
except (ValueError, KeyError, TypeError, OSError, RuntimeError) as exc:
|
||||
print(f'Player error: {exc}', file=sys.stderr)
|
||||
return 2
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
raise SystemExit(main())
|
||||
Reference in new issue
Block a user