We're building Chisel to solve this. Instead of wrestling with nvtx events and complex CUDA interfaces, you just add decorators to the functions you want to profile. As simple as:

app = ChiselApp("my-app", gpu=GPUType.A100_80GB_1)

@app.capture_trace(trace_name="gpu_task")

These 2 lines will get you the profiling trace for your code on an A100.

Run your code normally, then view results in a clean dashboard or locally.

Documentation: https://herdora.mintlify.app/

If you sign up this week, there's $50 in free credits to try it out (our profiling runs are billed by the second, so this is a ton of free profiling runs!)

We'd love feedback from anyone who gives it a shot. We're also currently developing a C++ SDK for teams that need to profile lower-level code. Let us know if this would be of interest to you. We want to learn what profiling pain points have you run into with ML codebases. Please reach out if you have any fun stories to tell :). We very much welcome contributions!

The github repo is: https://github.com/Herdora/chisel