Show HN: cunoFS – mount S3/AZ/GCP storage *without* FUSE, 60x faster than s3fs

19 points

2 years ago

Hello HN! We’re Dan and Vaughan, based in Cambridge, UK. We recently launched cunoFS, a filesystem for fast, POSIX object storage access. Yes, we’ve heard that “Object Storage is just not suitable for POSIX workloads” or that “workloads should be rewritten to natively support S3”. But what if it could be? And often much faster than S3-native apps? We’ve worked hard to make this possible, without the shortcomings of other approaches.

Summary here:

- Run (unmodified) applications with S3/AZ/GCP or “S3 compatible” storage, for example:

  ls s3:// az:// gs://
  ffmpeg -i s3://bkt/vid.mov -c copy s3://bkt/vid.mp4
  rsync --inplace -Wav $(pwd) az://myaz/bucket

  auto fd = open("az://myaz/bucket/file.mp4", O_RDWR);
  read(fd, buf, 1024);

- POSIX-compliant: supports symlinks, hardlinks (emulated), UIDs & GIDs, permissions, random writes, etc.

- No additional server or gateway needed. Each object is a file, each file is an object directly accessible on native S3/AZ/GCP. No scrambling of data into a funky backend format.

- Bypass FUSE and maximise performance with our ultra-fast syscall interception engine (Dynasaur [2]), or optionally run as a FUSE filesystem

- Super fast - for many benchmarks it beats EFS, FSx Lustre or even attached EBS

  - Benchmarked at 50+ Gbps per EC2 node ⇔ AWS S3, scaling to 10+ Tbps [1]

  - 60x faster than s3fs in small file benchmark (copying Linux kernel source files)

  - PyTorch dataset reads boosted up to 13x (with accelerator package installed)

- Supports virtual mount points, FUSE mount points, and direct URI access.

Pricing: cunoFS is free for personal use. You need to register for a free license after 14 days. For commercial use, we offer a 30-day free eval license.

Check out our docs [3], or download now from our website [4].

P.S. Calling for partnerships/collaborations: We’re thinking of opening up Dynasaur, our ultra-fast syscall interception engine [2]. It covers all binaries (dynamic, static and semi-static) and runs in unprivileged containers/serverless. It can virtualise Linux kernel functionality and dynamically augment it with something else, without needing special privileges. Applications include user mode networking, live process migration, memory offloading, virtualisation, analytics, user-mode kernel experimentation, and of course running other FUSE file systems faster, without the FUSE bit. Please let us know if you are interested.

Looking forward to your feedback!

[1] https://cuno.io/technology/

[2] https://cuno.io/technology-details/#The_problem_with_SHIMs

[3] https://cuno-cunofs.readthedocs-hosted.com/en/stable/

[4] https://cuno.io/download/

4 comments

ls s3:// az:// gs:// ffmpeg -i s3://bkt/vid.mov -c copy s3://bkt/vid.mp4 rsync --inplace -Wav $(pwd) az://myaz/bucket auto fd = open("az://myaz/bucket/file.mp4", O_RDWR); read(fd, buf, 1024);

- Benchmarked at 50+ Gbps per EC2 node ⇔ AWS S3, scaling to 10+ Tbps [1] - 60x faster than s3fs in small file benchmark (copying Linux kernel source files) - PyTorch dataset reads boosted up to 13x (with accelerator package installed)