Show HN: Swim gossip protocol in Rust with mio/epoll

1 point

5 months ago

I built this to understand how epoll works under the hood and why it matters for networked systems.

SWIM is a gossip protocol for failure detection in distributed systems - each node periodically pings random members and if they don't respond, asks other nodes to try (indirect probing). Simple but effective.

  What I learned about epoll:                                                                                             
                                                                                                                          
  1. It's O(1) vs poll/select's O(n). With 10k connections, poll scans all of them every call. epoll only returns the ready ones.                                                                                                             
                                                                                                                          
  2. When idle, epoll_wait just sleeps - zero CPU. The kernel wakes you only when data arrives. In my traces:             
                                                                                                                          
     epoll_wait = 1  <10µs>     ← data ready                                                                              
     sendto      <52µs>    ← send                                                                                         
     recvfrom    <14µs>    ← receive                                                                                      
     epoll_wait = 0  <1.001s>   ← sleeps 1 second, 0% CPU                                                                 
                                                                                                                          
  3. Edge-triggered mode is tricky. mio uses it by default, so you won't get re-notified for writability until you hit    
  EWOULDBLOCK.                                                                 
                                                                                                                          
  Happy to discuss the implementation or answer questions about epoll gotchas.

What I learned about epoll: 1. It's O(1) vs poll/select's O(n). With 10k connections, poll scans all of them every call. epoll only returns the ready ones. 2. When idle, epoll_wait just sleeps - zero CPU. The kernel wakes you only when data arrives. In my traces: epoll_wait = 1 <10µs> ← data ready sendto <52µs> ← send recvfrom <14µs> ← receive epoll_wait = 0 <1.001s> ← sleeps 1 second, 0% CPU 3. Edge-triggered mode is tricky. mio uses it by default, so you won't get re-notified for writability until you hit EWOULDBLOCK. Happy to discuss the implementation or answer questions about epoll gotchas.