2024-04-22

Meerkat build log

by mail@adihegde.com (Aditya Hegde)

WIP tech log go distributed networking progress

Meerkat is a p2p distributed application to maintain data synchronisation between all nodes using GRPC. Initially part of a hackathon idea proposed (available in this post)

Commits

• 3f8b029 -> Migrating to GRPC and testing out basic node pinging.
• 2291e07 -> Implement a basic pool joining protocol.
• e9da45e -> Data transmission with failure in edge cases. Using 2 go routines to handle different aspects of the program.
• 7a51506 -> Creating mutex locks to fix data race issues persistent in the program.
• 6acc241 -> Basic graceful exit for simple fault tolerance during signal interrupts
• b8ffb9f -> Creating Disk Snapshots to identify events of DELETION, ADDITION (updates and addition are broadcasted as the same type of event for now).

Issues

• Issue #7: Streaming file contents as chunks
  • Current method passes all bytes in one go through a stream. This is an issue with larger files
• Issue #6: fix: file deletion broadcasting
  • file deletions
  • folder operations are shadowed
• Issue #5: _ Handle unexpected disconnects from nodes_
  • Handle interrupts
  • Handle fatal erros
• Issue #4: Dealing with merging 2 network pools
  • Focus on this issue is to work with network partitions.
  • Current system only deals with how a node can enter an existing network pool.
• Issue #3: Move towards a gossip protocol
  • Broadcast of changes happes to all nodes
  • Move to gossip to reduce network overload, the goal is that data propagates to all nodes eventually.
• Issue #2: Handshakes and Disconnects not going through
  • Duplicate addition of nodes in the client list
  • Safe node leaving boradcast doesn't propagate across the nodes
• Issue #1: Attempt to connect to client twice

Research for future improvements

• Raft Consensus Algorithm Paper
• Distributed Systems lecture by Martin Kleppmann