Not an issue if downgrading to chart.js 4.3.0. I guess this means it's a chart.js problem. See this comment.

I have same issue: https://stackblitz.com/edit/stackblitz-starters-ngv2t8?file=src%2Fapp%2Fapp.component.ts

Bug happens with both wheel and drag and on all axis modes. I have linear scales and no limits. Downgrading to Chartjs 4.3.0 fixed it.

I also noticed that same issue happens in chartjs-plugin-zoom Click-to-zoom sample (only sample with linear scale): https://www.chartjs.org/chartjs-plugin-zoom/latest/samples/wheel/click-zoom.html, even though it says be using ChartJs 4.2.1.

Same issue. Seems like problem with >4.3.1. 4.3.0 (last update 3 month ago) works for me