With the help of some pprof, I was able to trace it to a serious performance regression introduced two weeks ago, and come up with the fix (happy to field questions, if you're interested in the nitty gritty).
It's not the fastest driver around, but it's no longer the slowest: comfortably middle of the pack. It's based on a WASM build of SQLite, and thanks to https://wazero.io doesn't need CGO.