GoBlr #86 30 May 2026

Aditya Hegde — Sat, 30 May 2026 00:00:00 +0000

Go routine and channel internals

Understanding the Go Runtime: The Bootstrap internals-for-interns.com
Paper: Communicating Sequential Processes www.cs.cmu.edu
Scalable Go Scheduler Design Doc by Dmitry (2012) docs.google.com
Go's Concurrency and Channel Internals blog.bullgare.com
Scheduling In Go : Part I - OS Scheduler www.ardanlabs.com
Go Scheduler nghiant3223.github.io
Memory Allocation in Go nghiant3223.github.io
Goroutine Scheduler, go-profile notes datadoghq.dev
Go Network Programming: How Epoll Powers Performance binarymusings.org
Scaling Go to 192 cores for Heavy IO jazco.dev
GOMAXPROCS, epoll/kqueue, and Scheduler-Level Tuning goperf.dev
creation of the hchan struct at channel creation. A heap allocated circular buffer protected by a mutex. sendx and recvx. The reference to this hchan struct is maintained in the goroutine stack
We have a sendq and a recvq that are doubly linked lists. The data type is a waitq. The buffer points to a circular queue. The each data type in this LL is a sudog type and has an unsafe pointer to an element. This is the memory address of a variable. This also holds g type which is a reference object or the identity of the go routine holding data about waiting, the status of the routine

Buffered channel

our circular queue is the buf where we write to the circular queue
ch <- 3 we push a value into the channel. In the case when its full, we create a sudog object as we cannot write to the channel buffer. This sudog object is placed in the sendq DLL and this is our queue for items that are trying to be written to the buffer.
The go routine calls gopark and the go scheduler changes the state of the go routine to waiting.
result <- ch will read from the buf circular queue.

ch <- 1
ch <- 2
ch <- 3 // this is pushed to sendq
	res <- ch // 1	
// sendq value of 3 pushed into queue
	res <- ch // 2
	res <- ch // 3
	res <- ch // we now call gopark as there is nothign in the buf. We have a sudog allocated and puhsed into the recvq

when a sender now publishes an element into the channel, instead of writing to the buf we directly do a memcopy() to the recv. We directly copy to goroutine stack and we do not touch the circular queue buf

func recv(c *hchan, sg *sudog, ep unsafe.Pointer, unlockf func(), skip int) {
  // snip
  if c.dataqsiz == 0 {
    if raceenabled {
      racesync(c, sg)
    }
    if ep != nil {
      // copy data from sender
      recvDirect(c.elemtype, sg, ep)
    }
    // snip
  }
  // snip
}

func recvDirect(t *_type, sg *sudog, dst unsafe.Pointer) {
  // dst is on our stack or the heap, src is on another stack.
  // The channel is locked, so src will not move during this
  // operation.
  src := sg.elem.get()
  typeBitsBulkBarrier(t, uintptr(dst), uintptr(src), t.Size_)
  memmove(dst, src, t.Size_)
}

Unbuffered channel flow

we have direct stack to stack copy. We do not have a circular queue holding elements being sent

Select statement

select {
  case <-ch1:
  case <-ch2:
  default: // default non blocking exit
}

select { // blocking multi-select
  case <-ch1:
  case <-ch2:
}

Patterns

actor-styled ownership pattern
shared state pattern
worker pool pattern for bounded concurrency
fan in and out
How Go (Golang) Works — A Deep Dive into Runtime Internals muratdemirci.com.tr
Go Concurrency Patterns: Context go.dev
Go Concurrency Patterns: Pipelines and cancellation go.dev
developer.hashicorp.com rollback manager for cleanup
Leaking goroutines by Karl Seguin openmymind.net
Goroutine Leaks - The Forgotten Sender ardanlabs.com
Proposal: Goroutine leak detection via garbage collection go.googlesource.com
boundary hashicorp architecture
- Controller/Worker Model www.ibm.com
- Workers concept in boundary developer.hashicorp.com
Control plane vs. data plane www.ibm.com
Different planes (control, data and management plane) in systems developer.hashicorp.com
- Having a separate plane for blast radius, network reach, horz scaling, upgrade cadence, multi tenancy of networks
- Overkill when we have single tenant, stateless apis, when latency to control plane is acceptable
Designing resilient systems by IBM and Hashicorp developer.hashicorp.com

dbw by hashicorp (https://github.com/hashicorp/go-dbw) as a database wrapper in boundary

boundary uses a public key infra (PKI) www.hashicorp.com
Mutual TLS www.cloudflare.com
What is Mutual Authentication www.cloudflare.com
More on Zero Trust security www.cloudflare.com
Atomic pointers in golang pkg.go.dev
Atomic pointer operations and unsafe groups.google.com
Go sync.Map: The Right Tool for the Right Job victoriametrics.com
io.Copy in goroutine to prevent blocking stackoverflow.com
Goroutines, Nonblocking I/O, And Memory Usage eklitzke.org
Chapter 8. Goroutines and Channels - Shichao's Notes notes.shichao.io
Go I/O Readers, Writers, and Data in Motion victoriametrics.com
gRPC-Gateway grpc-ecosystem.github.io

Rust Bangalore Meetup 16 May 2026

Aditya Hegde — Sat, 16 May 2026 00:00:00 +0000

Talk on Async Runtimes and async I/O in Linux by Sanchayan Maity. Slides for the talk.
Asynchronous I/O programming with io_uring unixism.net
Async IO on Linux: select, poll, epoll jvns.ca
Notes on epoll and io_uring - OS readiness vs OS completion models iafisher.com
Notes on io_uring boats.gitlab.io
Asynchronous IO fundamentals www.ncameron.org
The Impact of Thread-Per-Core Architecture on Application Tail Latency penberg.org
Thread-per core without.boats
Io Uring nick-black.com
I/O access and interrupts linux-kernel-labs.github.io
Compio - a thread-per-core async runtime with io_uring github.com/compio-rs
Compio with executor and dispatcher decoupled that can be swapped compio.rs
Scheduling multithreaded computations by work stealing dl.acm.org
Completion model makes use of an ownership model. There can be an issue with respect to drop problem on how buffer should be handled when a future is dropped as results will be written back to this buffer. (I' not sure about this, have to research). Complexities wrt in flight future operation cancellations. Dropping future does not stop kernels handling of the memory that it was given ownership to.

// compio (ownership mode)
// kernel must have ownership to the buffer that it is writing to
let buf = [0; 4096];
// buf is not borrowed
let (res, buf) = file.read_at(buf, 0).await;
//        ^^^ buffer returned with data

// tokio (borrowing) model
file.read(&mut buf).await?; // here buf is borrowed

Advantages wrt zero-copy optimisations, completion models support batched submission for I/O operations, etc (check slides),
Apache Iggy benchmarking benchmarks.iggy.apache.org
Apache Iggy: Hyper efficient Message Streaming github.com/apache

Platform Engineering Meetup 9 May 2026

Aditya Hegde — Sat, 09 May 2026 00:00:00 +0000

Guava core lib. concurrent Rate Limiter guava.dev
Clockwork: The Backbone of PhonePe’s 2 Billion Daily Jobs tech.phonepe.com
Temporal (a scalable and reliable runtime for durable function executions) Docs docs.temporal.io
Drove - Distributed Container Orchestrator by PhonePe slide deck
Dover Container Orchestrator docs dover
Kubernetes Event-driven Autoscaling keda.sh
Oracle ScheduledExecutorService interface docs.oracle.com
RabbitMQ docs on Queues www.rabbitmq.com

Bengaluru Systems Meetup 25 April 2026

Aditya Hegde — Sat, 25 Apr 2026 00:00:00 +0000

Redis/Valkey Replication Internals: The Architecture Behind Zero-Copy Command Propagation by Sourav Singh Rawat frostzt.com
pgrx - a framework for developing PostgreSQL extensions in Rust github.com/pgcentralfoundation
Tantivy is a full-text search engine library inspired by Apache Lucene and written in Rust github.com/quickwit-oss
BM25 index
- BM25 article arpitbhayani.me
- What is BM25? paradedb.com
- Okapi BM25 en.wikipedia.org
Parade: Simple, Elastic-quality search for Postgres github.com/paradedb

Boehm’s curve in life

Aditya Hegde — Wed, 31 Dec 2025 00:00:00 +0000

This was something I had in mind to write down a year ago, but now that I think of it, that wouldn’t have mattered as I believe that it was this year that has taught me more lessons in life than any other up to this point. It’s really funny how fast life can change and how close things really are.

The current position that I am in is fully because of me turning a blind eye to multiple situations, and not having the courage to stand up for myself. And to be transparent, I have never learnt how to give myself respect. For years, I’ve always tried to gain that respect by doing a favour for others, without ever thinking about positioning myself at the same level. Self-respect is something that I’ve lacked which has lead to some poor decisions.

It is necessary and true that all of the things we say in science, all of the conclusions, are uncertain, because they are only conclusions. They are guesses as to what is going to happen, and you cannot know what will happen, because you have not made the most complete experiments.

Richard Feynman

Looking at the position I was in about 8-10 months back, I probably would have never expected it to spiral out in the manner it has. If I were to draw out a comparison of the person that I was a year ago, my self esteem, my views on relationships and friendships, my ideal lifestyle, all that seems so different. I always had these idealistic scenarios running in my head for situations to calm myself down and feed into what would be candy to my brain at the cost of hurting others.

You can not have the cake and eat it too

I really don’t think I would have been the same person that I’ve tried to grow into today if I sat idle during my summer break. It’s concerning how fast my thoughts back then could take me away to a place that forces me to be someone that isn’t myself. It’s really the people with whom you decide to surround yourself with that mould you into another person, and the people I met over this entire year has taught me one thing - I’ve driven my life to shit with my previous decisions.

With these decisions, I’ve hurt a couple of people close to me. I tried standing neutral to arguments thinking that I can be on the “good side” of others and life will get back to what I presumed was stable. It’s never worth it, I made that mistake twice in the same year and I’m still grateful for having a second chance at realising and admitting to my mistakes. I consciously went ahead with things knowing the possible consequences to it, and went ahead to ensure that I don’t ruin something that I thought I had. While I’ve come to realise them, as a consequence, I’ve lost out on precious time with others.

Listen to others, you are probably missing out on a greater perspective of a problem than you might think. If you surround yourself with the right people, they will always be your anchor to reality. If you choose to ignore those people for others in the name of keeping everything in your life in symmetry, you not only hurt them, you hurt yourself. At one point of the year, all those values that I collected were tossed in a bin to tell myself, “At least someone else is happy”... It’s disrespectful to yourself and the person who gave their time and effort to a problem that isn’t theirs.

Time is never on your side. If you trick yourself into thinking it is to keep yourself calm, then you’ve just ruined the chance for something fruitful to happen. It’s weird that I’ve never really realised how limited time is, and everything just seems to fly by. Turning 21 has made me realise...damn I don’t have much time left before I have to settle down in life. It was only 5 years ago that my mother told me as I entered the 11th grade - the next years till you complete your school and college will happen in an instant, and I’ve come to realise now, which feels too late. Holding onto those mistakes without correcting them, shutting yourself down that you will rectify them soon when then noise around you has settled down, you won’t even come to realise the time that you have lost. Seriously, just be straight and apologise. It’s the easiest thing to do, and the best way to get rid of all that stress and guilt that has been collecting up.

There are only a couple of people who I can really talk to, being myself, who value honesty and share the same values. These are the people who are your actual friends. Yes, sounds pretty obvious, but when you make the mistake of wearing a mask to please others, you drift apart from these people and hurt them. Learn to burn bridges, sometimes they set you free from a you being a prisoner to your idealistic view of the world. Talk to these people, open up to them, and give them your time as well. Life moves fast, you don’t really know how much more time you have to before you move to a different phase of your life where people have to go in different directions. When you keep yourself surrounded by these type of people, you grow with them and in the right direction. You begin to realise the mistakes that you have made previously and you begin to correct them. It’s never too late to realise them, but sooner the better. Don’t wait for your life to cross paths with others later on to correct your mistake.

Lost a bit of spark

Life has been draining. This year has had a lot of personal issues spew up that threw me off guard, consequences of the mistakes I’ve made, assignments and submissions part of my undergrad has almost pushed me off the edge of my sanity. It made me hit the lowest part of my life to come to realise what I’ve done wrong. It’s still embarrassing that my proudest achievements are all back from my first year of my college. The amount of work from college and my lack of interest with the quality of education, made me use it as an excuse for not working and let myself rot into being unproductive. I let this take the better of me and haven’t had anything proud to showcase that I accomplished in my career.

Moving into 2026, I need a reset back to when I joined college. With what is essentially my final semester of college, I need to treat it like my first year. I don’t have a job, nothing to back me up with. Make use of that time to learn as much as I can, not aimlessly, but with the same intensions and curiosity that I had back when it all started. It feels cliche to say that, but It’s just a reminder to myself that, I managed to accomplish more with that mindset and I cannot rely on my work from 2 years ago to define my career. While I do have great learnings and outcomes from my internship and several instances during the year, I have not been loyal to myself in learning more and understanding to the depth that I used to try to back then.

Things to remember

I’m glad that I’ve come to realise errors in the way I’ve valued my life. Cutting the unnecessary dead-weights out before entering a new year is giving me a cleaner slate to work with. Knowing that I’m surrounded by the right crowd is already a positive step into the unknown next year. Just being more transparent about my mistakes and being more respectful to myself by itself has let my life incrementally move closer back to a better place. I’m no way close to where I should be, but all these experiences has taught me things the hard way.

I've never actually reflected back on a year, probably doing it now because i've come to realise a lot of mistakes i've made on my end, and a lot of things that have just bottled up across the year.

Bengaluru Systems 11

Aditya Hegde — Sat, 06 Sep 2025 00:00:00 +0000

Overview

For those who aren't aware Bengaluru Systems, is a really cool community of systems enthusiasts here in Bengaluru, India. The meetups cover insightful talks on distributed systems, databases, compilers, and more! It was created back in May 2024 by some amazing folks like Madhav and Rowjee.

This edition introduced a lightning talk format where all of us were given the opportunity to present what we have been working on or what we found interesting.

This event saw several lightning talks including ones given by Navin (who spoke about his Torrent client in Rust) and Sudhir who spoke about his findings and research on Swiss Tables, which are used by Go 1.24, Valkey, and Rust’s port of Google's Swiss table hashbrown.

My talk on Quaso

During the event, I was given the opportunity to present an old project of mine - Quaso (a Raft implementation in Rust). During this presentation, I gave a basic overview of how Raft leader election works and how new entries to a log are replicated across a cluster. The presentation concluded with a short demo on leader elections and re-elections, simulating append entries, and a demo with a basic Key-Value store.

After this, we had @nilderef talk about his implementation of Raft in rust + how the raft protocol works in general. We also saw a very cool live demo of leader election and simulated failures, which was a very cool demonstration of fault tolerance in real life. pic.twitter.com/RxixiiCXV0
— Bengaluru Systems (fka Bengaluru Systems Meetup) (@BengaluruSys) September 17, 2025

I spoke about how my current implementation works, how it can be extended to have consensus over any serializable type in Rust. I touched upon my previous implementation which made use of Google's tarpc crate in Rust, and the issues that came up with it.

This was my first individual talk, and my first in a while (the last one being my presentation for anna at GoLang Bengaluru #76).

Images

taken by @anirudhsudhir

taken by @anirudhrowjee

Presentation

Twitter Post

After this, we had @nilderef talk about his implementation of Raft in rust + how the raft protocol works in general. We also saw a very cool live demo of leader election and simulated failures, which was a very cool demonstration of fault tolerance in real life. pic.twitter.com/RxixiiCXV0
— Bengaluru Systems (fka Bengaluru Systems Meetup) (@BengaluruSys) September 17, 2025

Pes Innovation Lab

Aditya Hegde — Mon, 09 Jun 2025 00:00:00 +0000

Interned at PIL (aka. PES Innovation Lab (prev. Microsoft Innovation Lab (prev. Nokia Innovation Lab))) for 8 weeks during the Summer of 2025. This is a research lab based of in the main PESU Campus. The problem statements we choose to work on are created by seniors and alumni from the lab. My research involved introducing a preemptive task scheduler to MCUs(Microcontroller Units). Our scheduler is built around the Raspberry Pi Pico, which comes with a dual core ARM-Cortex M0+ processor. Our implementation introduces a novel approach in loading compiled programs at runtime via a serial interactive interface which is designed to control the status of the tasks loaded on the board.

This project involved several blockers that took weeks to tackle, such as implementing preemptive context switching (which would be the foundation to the entire "operating system") proved to be a challenging task. Will be updating more on the implementation when I get the time.

These 8 weeks has definitely been an experience to say the least. Learnt several lessons as an outcome of this. This also came up in parallel to my first few weeks as being the part of the core team of HSP so managing my time between to things that I am held accountable to, while probably not being my first, has definitely been the bigger challenge for me. This is probably because our results took quite a lot of time compared to the other teams in the lab, which did add to my stress personally, but it was a process I enjoyed.

Picture taken with my lab batch and seniors on the last day dinner party :D

I purchased a new camera

Aditya Hegde — Mon, 21 Apr 2025 00:00:00 +0000

Recently purchased my first* camera - Sony α 6700. Theres an asterisk to that first as some of you may be familiar with my unsplash page or the fact that I have been taking pictures for a couple of years. Almost every previous camera that i've used was what my mother purchased back when I was a year old. In fact, my widely used Sony Rebel XSi is a year older than me. Hence this makes it my first ever camera.

Taken on my mom's Cannon PowerShot SD630, apples retouch tool coming in handy to remove my drivers license from the image!

Here are the details of the gear that I currently use

Camera Body: Sony ILCE-6700 APS-C
Lens: Sony E PZ G 18-105 mm F4 G Lens

First pictures on the SD card are my parents :D

This inked is dated to when I made this purchase. I have however been using the camera for the past 4 months (written on 28th Sep, 2025)

So far i've taken this camera with me for several trips, one of which was a short summer trip to Pondicherry with my college mates. Besides that i've sort of made it a regular thing to carry my camera around. I still find it difficult to take a camera out of my bag and start clicking pictures in the public. This is perhaps only because of how people in my country would perceive using a camera as "professional photography", and I will save my rant on this for another inked blog.

There are a shit ton of laws that are hidden, or supposed to be "known". I find these laws on cameras pointless and unfair, just seems like an incorrect opinion on photography in general. You are somehow allowed to take pictures with your phone (modern phones are more than capable of taking professional grade pictures), but it becomes a problem to use a device which is designed solely for this purpose. Have almost gotten my camera confiscated twice. Once at a metro station and another at a beach. I am very careful with when I make use of the camera and only try taking pictures in an appropriate environment. The same for probably when i make use of my phone.

I've never used a mirrorless camera before. Probably the one downside is loosing out on battery life compared to DSLR. I've only had to charge the old Cannon Rebel XSi once during a 10 day Singapore trip (you will probably find an inked blog on this soon, have lots of pictures from here on my unsplash page), with extensive usage. Same cant be said for mirrorless as, well theres no viewfinder (yes there is a viewfinder but it uses a screen, thats the whole point of mirrorless, no mirror).

Another upgrade with the camera is the lens. I've never used anything more than a 55mm lens on my older Cannon and i've gotta say, 105mm is really handy for a couple of things. Here is me taking pictures of an Air India eXpress plane doing rounds around my sisters place in Bengaluru

Yes theres a bit of digital zoom, but you can see how close you can get to the place in the little left box below!!

I'm no expert in these things, I struggle quite a bit in low-light conditions and am yet to truly understand how to color grade my pictures (@brendangleberries instagram page is probably what inspires me/the form of art that I like, but more focused towards photography than cinematography). I love doing this in my free time and trying to find ways that I can improve. Theres something different about the felling of taking pictures on these cameras. I love capturing the moment that I see before my eyes, and keep them forever (always have been known for hoarding memories with myself, my parents love complaining about this). And I hope to share more milestones that i've come across here with this little hobby of mine (I'm apparently in the top 25% of contributors in unsplash but thats probably not alot)

Shameless plug to @adihegde. Almost 400k views!!!

Anyways, I have a lot to share, i'll probably update this inked blog, or make a few more as and when I plan to share more cool pics or things I come across while learning (if I get the time, 3rd year is loaded with too many assignments). Thanks for reading this blog (and please do go like my posts on unsplash :D)

Studying db hexdumps

Aditya Hegde — Wed, 16 Apr 2025 00:00:00 +0000

This post is WIP. Based on my repo parsley.py

Building systems in rust

Aditya Hegde — Sat, 21 Dec 2024 00:00:00 +0000

blog updated 21/10/2024

A much needed Introduction to this Blog

As I said, this is work in progress. The stupid reality where I open source my progress and not the repository that I'm working on in shame, definitely working with a garage door open. This section is still 🏗️

How this project started out

A distributed bit-cask like LS Hash Table KV Store. Or according to my first, definitely not force pushed commit 4a141713 on Mon Sep 23 00:31:41 2024 +0530

snowflk is a distributed kv store inspired from bitcask's implementation of a Log-Structured Hash Table, focusing on replicated storage of persistent data across nodes

This is another shot of me dipping my toes in rust, which was not the initial plan. Go seemed easy, but I wanted to learn something new along the way as well. May not be a log of progress considering academics and club related work, wanted to try some stuff out :)

writing this log right before a test, might be messy

SEP 23

Genesis, started of with a go implementation but...want to try rust for no reason, lets see how that goes :P

SEP 24

Making this public, I'm to ashamed of what i have written

SEP 30

Implemented an abstraction over yaml_rust2 crate for parsing config.yml during startups
Plans for messagepack for persistence of states, i.e. for raft that would include data like current term, node votedFor and the log[] of committed changes
Writing tests for raft from scratch will be complicated. Plus, need to decide if I should be just using simple TCPListeners and move bytes around or try out Tonic for gRPC

Back at it again...changing everything

With everything basically scrapped, except that one beautiful yaml parser I wrote with the yaml_rust2 and learning a bit more after not finish advent of code once again, I felt it would be a great time to rewrite this for fun during my end semesters...which is still going on as I write this line.

So with a lot of undesirable behaviour, and exams this project had to come to a hold. Started the whole thing with a fresh repository and began working on a more generalised raft library. Some small changes included using bincode for serialization instead of message pack showing significantly faster serialization speeds than messagepack.

#[tarpc::service]
trait Server<T> {
    // ...
    fn ping_example(&self, req: AppendEntries<T>) -> Response;
    // ...
}

Having a generalized raft library meant serializing bytes and sending generic logs over TCP, which isn't really that hard. But tarpc didn't really like doing this as it never really allowed me to write generic traits for a server, which was of course needed. Instead of this, wouldn't it be easier to just, excuse my words, raw dog the bytes over tcp??

If I can write my own rpc like(definitely not RPC) layer then, I technically have all the flexibility I need to have, couldn't be that hard considering its all just bytes sent over network, and deserialization works if and only if the types on both the sides are correct. As of now, there is absolutely no type validation on client and server side besides the tokio::task that has the server running panics out of existence while the task of the application is still running. Of course not desirable but hey, this is still in a toy stage, wont hurt to experiment

Just to point out to those who have not seen my other f ups while doing something like this, the last time I tried writing a protocol with pure tcp and marshaled json bytes across a network did not end very nicely, in-fact never managed to send bytes across because go never warned me about private fields in a struct don't get serialized at all, thats really fun to learn about almost a year after a wasted hackathon, but hey you always learn something at the wrong times

Why all this, well lets see how elegantly you can now define a naive type in rust and have raft as a consensus layer

#[derive(Debug, Serialize, Deserialize)]
pub struct Pair<K, V>
where
    K: 'static + Eq + Hash + Debug + Send + Clone,
    V: 'static + Debug + Send + Clone,
{
    pub key: K,
    pub val: V,
}

// skipping the necessary implemnetations for Clone,
// Display, server::log::Entry (which is a necessasry
// trait of a generic T to be in consensus, more on
// that later)
#[derive(Debug)]
pub struct KVStore<K, V>
where
    K: 'static + Eq + Hash + Debug + Serialize + DeserializeOwned + Send + Clone,
    V: 'static + Debug + Serialize + DeserializeOwned + Send + Clone,
{
    pub map: Mutex<HashMap<K, V>>,
    pub raft: Raft<Pair<K, V>>,
}

Now isnt that neat, atleast to my eyes that has seen a lot worse, now all you need is some async function calls and viola!

#[tokio::main]
async fn main() {
    // some config loading
    let args = Args::parse();
    let config =
        parse_config(PathBuf::from(args.conf_path))
        .expect("failed to parse config");

    // A simple KVStore storing keys as strings and values as a string
    let kv: KVStore<String, String> =
        KVStore::init_from_conf(&config).await;

    kv.raft.start_raft_server().await;
    let kv_raft = Arc::new(Mutex::new(kv.raft));
    let kv_raft_clone = Arc::clone(&kv_raft);

    tokio::spawn(async move {
        kv_raft_clone.lock().await.tick().await;
    });

    // our listener for a kv server to listen to clients over
    // tcp
}

Okay now lets get back to what i've done to come to this stage. Now how did I define this layer...

#[derive(Debug, Deserialize, Serialize)]
pub enum RequestPattern<T>
where
    T: Entry + Debug + Display + Clone,
{
    PingRPC(PingRequest),
    AppendEntriesRPC(AppendEntriesRequest<T>),
    RequestVoteRPC(ElectionVoteRequest),
}


pub trait Service<T>
where
    T: Entry + Debug + Display + Clone,
{
    fn ping_node(
        node_id: NodeId,
        state: Arc<Mutex<State<T>>>,
        req: PingRequest,
    ) -> impl Future<Output = PingResponse>;
    fn append_entries(
        node_id: NodeId,
        state: Arc<Mutex<State<T>>>,
        req: AppendEntriesRequest<T>,
    ) -> impl Future<Output = AppendEntriesResponse>;
    fn request_vote(
        node_id: NodeId,
        state: Arc<Mutex<State<T>>>,
        req: ElectionVoteRequest,
    ) -> impl Future<Output = ElectionVoteResponse>;
}

#[derive(Debug, Clone)]
pub struct Server<T>
where
    // why `static, I basicaly enforce that T is always of an owned type,
    // otherwise the compiler runs in the assumption that T could be of type
    // &T, &&T, &mut T,... which is obviously never the case
    T: 'static Entry + Debug + Display + Serialize + DeserializeOwned + Clone,
{
    // i know this is very redundant, but I did face stupid
    // issues of my `State<T>` not implementing send across
    // threads which definitely whent over my head while
    // trying doing this the first time, so yeah its a bit
    // messy needs a lot of clean up
    pub state: Arc<Mutex<State<T>>>,
    pub node_id: NodeId,
}

Now lets have a look of what we can do on the server side for an incoming TCP request

if stream.read_buf(&mut buf).is_ok() {
    let req: RequestPattern<T> = bincode::deserialize(&buf).unwrap();

    match req {
        RequestPattern::PingRPC(req) => {
        	/* handle resp back here */
        },
        RequestPattern::RequestVoteRPC(req) => {
        	/* handle resp back here */
        },
        RequestPattern::AppendEntriesRPC(req) => {
        	/* handle resp back here */
        },
    }
} else {
  warn!("failed to read stream {:?}", stream);
}

I also went ahead and made some changes on how the raft state is defined, separating its volatile component from the persistent state

#[derive(Debug)]
pub struct State<T: Entry + Debug + Display> {
    pub persistent_state: PersistentState<T>,
    pub volatile_state: VolatileState,
    pub recieved_leader_heartbeat: AtomicBool,
}

#[derive(Debug)]
pub struct Raft<T>
where
    T: Entry + Debug + Display + Serialize + DeserializeOwned + Clone,
{
    pub node_id: NodeId,
    pub config: Config,
    pub state: Arc<Mutex<State<T>>>,

    // termination condition for the
    // server ticker
    pub stopped: bool,

    // for the "rpc"
    pub server: Server<T>,
    pub client: Client,
}

And for anything that I missed out, here is the long commit message that I mostly typed (and definitely have not force pushed to a lot) while I was incredibly sleepy...yeah I've gone against all the best practices so far, especially with respect to my commits

(17/12/24) NOTE: A lot of un deterministic behaviour as it stands. Append Entries only sends empty entries to all nodes, just for the sake of testing for now. The cluster remains stable up till a point, and after that all nodes start fighting over leadership like children. Need to read the TLA spec for raft and the paper carefully. Good progress so far

21/12/2024 Almost There

commit 462a4be4
Author: bwaklog <aditya.mh@outlook.com>
Date:   Sat Dec 21 02:01:25 2024 +0530

feat: Leader Election

So small update, finally finished my exams and have gotten back at this. And looks like we finally have some sort of a leader election working along with replication logs for heart-beats as I don't really have a working layer to take in commands from a client...still working on that part and should be up in the next commit

Now there isn't really a proper testing solution for this yet, but the cluster shows deterministic behaviour. I still believe its not fully up-to the original spec, but that should not take much of a change.

Reason for the previous undefined behaviour

The volatile state on server did not hold a separate timer for heartbeats. Each randomised interval for a election_timeout was from the range $[1, 8]$. The main tick() method for the raft state kept sleeping for a different randomised time in the range $[1, 8]$, a stupid mistake on my side which made the leader node wait for probably a longer amount of time than what was required.

Hence follower nodes that slept for a shorter duration are highly likely to have not recieved a heartbeat from the leader node which force them to move to a candidate state. $t_{\text{heartbeat}} << t_{\text{election timeout}}$. I still need to pick a proper ratio to separate out these times which is currently split as $[1, 4)$ for $t_{\text{heartbeat}}$ and $[4, 8]$ for $t_{\text{election timeout}}$ which does cause some issues. The MIT 6.5840 Spring 2024 labs do specify a proportion of $1:20 :: t_{\text{heartbeat}}:t_{\text{election timeout}}$, which i haven't really chosen for the time being. If the max duration for an election timeout were to be $10s$ that would make the max duration for a heartbeat $0.5s$ ish...

Now here comes part two of this problem which went over my head. When a server recieves an AppendEntriesRPC request, the first condition check is the comparing the node terms. A server has to reject an append rpc if the request term is lower than the current server term, the condition missing was that it was willing to reject a request regardless of the NodeRole. So a candidate that goes haywire would keep going to a higher term whilst receiving heartbeats from a Leader in a lower term.

Another minor mistake was not resetting the election_timeout timer and recieved_heartbeat boolean value that are a condition for a follower to candidate transition right after voting. Normally you send an empty heartbeat to all followers after receiving a quorum of votes, but as the heartbeats are sent out more frequently and wont give a chance for a server (i.e. a follower) to have its election timer to timeout, no un-necessary transitions will take place. I still feel that i'm overlooking this issue, and it can come and bite me in the back, but it should be a simple fix.

Up next

With most of this out of the way, I can start giving some time to working on the KV layer of this project. I want to focus on the internals of databases, so thats a small diversion I don't want to take for the time being. Besides that, I do need a working client to append to this raft log, so the KV will require some changes.

The next challenge lies in defining a proper message delivery system for committed raft entries. How do I deliver an entry to the application once consensus has been reached.

The goal of this is to have the consensus layer function as a asynchronous worker thread on a server that acts as a middleman for applying updates to state of an application.

When tries to change state of an application, it is first passed onto the consensus layer, so it is the responsibility of the consensus layer to pass this down once it has finished its task of replication as we don't want the KV to waste time in polling through the log of raft entries and find which of the log entries are freshly committed and can apply it to the state of its application. Such a system needs to work asynchronously in the background

24/12/2024 1:30am

Small update before heading into a Christmas vacation. Partially got log replication to work, no leader commits have been implemented. So what do I mean by partially? I have not fully tested this yet, this is just a very early commit so I don't loose my progress.

You might want to zoom into that image for a better look.

In the above image, the third horizontal pane is the only leader in the cluster which has taken set operations in the order set foo baz, set foo bar, set apple red. Do note that there is no current compaction RPC

WIP: Insert some hand drawn images here explaining replication

The mistake that I made was that I have been following the concept of the base index of being 1 only in some places, because of this mismatch of logic, lots of things went wrong. While basic replication works, I still haven't tested out replication in leadership changes in a cluster. Ironically have lesser time to work on this during a vacation than my endsems 😵‍💫

Whats wrong currently? So a node is able to continue replication even after re-election. But the problem comes when a node disconnects from a cluster. As persistence is somewhat broken at the time being and since a leader expects that all the logs are up to date, a node starting back after a crash will have a completely empty set of logs. What is to be done is, incase if a node rejects an entry, we need to decrease the next_index on the leaders side and keep retrying until we can successfully replicate the log.

If AppendEntries fails because of log inconsistency: decrement nextIndex and retry (§5.3)

24/12/2024 8:20pm

Clocking in just before my break, and finally have some good log replication. Took a few minutes to change all the indices to be logically based off with index 1. Don't fully like what i've written but i'll focus on optimizing this later.

Log replication commit ce3fc164

In the above image:

Cluster elects node $32$ as the leader for term $1$. Client sends the following commands set foo bar right before the server is manually stopped and restarted
Cluster elects node $64$ as its leader for term $2$. Client sends set bar baz which is replicated. The server is stopped and restarted
Cluster elects $32$ for term 3 after which the client sends set baz foo

commit ce3fc164f08aee7ce997aee1a66e00de22cb6506 (HEAD -> main, origin/main)
Author: bwaklog <aditya.mh@outlook.com>
Date:   Tue Dec 24 20:27:07 2024 +0530

    feat: Log replication

    Pending
    - log replication when client has to rewrite over conflicting logs
    - write leader commit logic
    - handle a delivery callback to update the underlying state

24/12/2024 11:07pm

Finally can wrap a part of this up. Persistence is working (commit 837db316), so is log replication and leader election. Whats left is leader commits which should not be hard. The tricky part remaining is the deliver function callbacks. Good wrap up before Christmas eve 🎄

Here is a small demo i've uploaded: link

04/01/2025

commit 6a123051e70c2e3c58bc2d30ee8a72b7c2713cd1
Author: bwaklog <aditya.mh@outlook.com>
Date:   Sat Jan 4 13:15:35 2025 +0530

    feat(wip): Leader Commits

Also fixed state persistence where I was initially persisting the entire log length to the disk

08/01/2025

commit 3b26e5487cabe65290d25630cba5257e0a2ada95
Author: bwaklog <aditya.mh@outlook.com>
Date:   Wed Jan 8 21:37:54 2025 +0530

    feat: Message delivery to underlying application

This required a bit on how the key value store functions, which needed a change to work with multiple background tasks and also with how the ownership model worked. The solution was basically threads and tokio::select! is super helpful.

What is needed is:

Raft needs to "deliver" messages of a newly committed log to the underlying state

A separate channel to handle message delivery from incoming clients. This isnt really a decision on how the raft struct is architecture, but does have some influence on how it can be used

Now on the kv store side, not really optimized but here is a simple switch between a client and raft channel

pub async fn generic_handler_interface(&mut self) {
	let deliver_rx = Arc::clone(&self.deliver_rx);
	let mut deliver_rx = deliver_rx.lock().await;

	let client_rx = Arc::clone(&self.client_rx);
	let mut client_rx = client_rx.lock().await;

	loop {
		tokio::select! {
			Some(client_message) = client_rx.recv() => {
				self.handle_client(client_message).await;
			}
			Some(raft_message) = deliver_rx.recv() => {
				self.apply(raft_message).await;
			}
		}
	}
}

Wrapping Up

The changes made were now to accommodate the previously mentioned generic handler on the KV store side which selects between a TCP client request or a message delivery via the deliver receiver. The raft state holds the transmitter and keeps pushing down this channel whenever a new log index is committed

if min_match_index > current_commit {
    for i in current_commit..min_match_index.0 {
      let deliver_tx = self.deliver_tx.lock().await;
       let _ = deliver_tx
					       .send(state.persistent_state.log[i].clone());
    }

    state.volatile_state.commited_index = min_match_index
			    .clone()
			    .to_owned();
}

The min_match_index here refers to the largest log index on all peers/followers till which, all previous log indexes are acknowledged and ready to be committed. If larger, the state machine delivers via the channel all the messages from the previous commit index on the leader up-to the match index. Once all delivered the commit index shifts updates itself.

The same logic goes for a follower receiving AppendEntriesRequest. The followers must ensure that it discards log entries when its current log length has a mismatch with a verified leaders log length and extend its log by extending its log with the new logs part of the leader request. If the leader’s commit index has shifted ahead of the followers commit index, the follower delivers each leader committed message (which was un delivered on the followers side) to the underlying application.

QOL

To make testing and getting a cluster up easier, all I’ve managed to move the binary to a docker image and run the raft cluster as a network of containers. Another size optimization I’ve learnt as of lately for docker images is having a multistage build bringing the image down to 54MB

docker pull bwaklog/quaso:latest
docker run \
	--rm -it \
	--network bridge \
	--privileged \
	bwaklog/quaso:latest

References

Repository: bwaklog/quaso

In Search of an Understandable Consensus Algorithm (Extended Version): paper link
Phil Eaton’s raft implementation github

I’d also like to thank @anirudhrowjee and @anirudhsudhir who i’ve been in discussions with during this implementation

Goland Bangalore 76

Aditya Hegde — Sat, 06 Jul 2024 00:00:00 +0000

Anirudh, Adhesh, Nathan and I recently got the oppourtunity to present our static site generator anna at the Goland bangalore meetup #76

Lunch with a Pigeon

Aditya Hegde — Thu, 04 Jul 2024 00:00:00 +0000

You aren't allowed to learn who you want aspire to become. Sometimes "experience" is more important even if that means to loose site of yourself

I'm writing this while sitting all alone in a deserted lab. Where I'm supposed to be interning and simulating some claims made by a paper. And I'm lost. I've taken this up only because I missed out on other chances for internships, and I'm afraid I won't get another shot at them either. And here I am sitting in this lab, and am not able to figure out a single thing. I don't have a grasp of this depth of knowledge in physics. I don't have a clue of what I'm doing

And now, I'm just in a lab and doing unproductive work. And why, well I'm told its for "experience" to add in that resume. Now I would be eating my own words if this is some internship that I wanted. But it isn't. I want to learn so many things, and I have just discovered something I'm interested in, but at the same time I've missed out on something to fill up in my resume. And out of guilt, I've ended up in a lab...a physics lab.

It's probably the 100th time I've evaluated what I'm missing out when I sit here. I don't get to expand my knowledge in the field that concerns me. Instead I'm tied down to a chair and doing something else. While I could have taken my summer at my own pace and covered everything, I've lost the luxury of social interaction and doing what I want. And it's getting frustrating. I'm not getting a benefit for myself, and I feel like I'm letting others down.

Techstack

Aditya Hegde — Mon, 10 Jun 2024 00:00:00 +0000

I've never had my self working on solely the terminal before. It would be only a few occasions when the GUI alternative was simply frustrating to use. I've since forced myself to try and venture more, and now I simply can't work without them. I have even lost count for the number of times i've used vim motions accidentally.

At first, yes these things were a bit intriguing, but once you leave the learning curve, everything just falls in place. You can find my dotfiles here

Text Editor

This is something that keeps changing once in a while, I like trying new stuff out there. But I always find myself getting back to neovim. While it took a while to get past that learning curve, I find it quite easy to use, and the motions are something I cant live without.

Zed is another editor I always find myself on. It supports vim motions out of the box, and the editor itself just insanely fast. And can't go without mentioning the amazing collaboration tool that it offers for free. Been trying emacs recently, and all i've experienced is frustration. It just has everything, and is a pain to use at times.

xkcd comic 378 - real programmers

Window Manager

Well first of all I use a Mac. Nothing comes along with it, not even window snapping. So organisation just gets worse as you use it. At first, I got myself to use this app called Rectangle, which provided some neat keybindings to snap windows to a position, but none of it was dynamic.

While I tried out yabai, which is lightweight and probably the most popular one out there, I just felt lazy to maintain its config. Thats when I discovered Amethyst, which I must say is insanely good. And have been using it for almost 2 complete semesters. There wasn't any need for setting up a config from scratch and making keybindings with skhd (which I only use to make global hotkeys to open some specific apps) which solves my problem.

Terminal Emulator

When I first started out on Mac, I didnt use anything but the default terminal application, which was restrictive in terms of customisation and basic colour display. iTerm solved most of those solutions for me, but even it couldn't display those nerd symbols without some weird cuts in the text. The scaling of some of the text and others in neovim weren't great.

Thats when I was introduced to GPU accelerated terminal emulators. Kitty, Alacritty, and as of now WezTerm.

Kitty has been great. Probably the most feature rich of the three. However, ever since I've messed up the config, I made my way to use wezterm.

So far, there hasn't been any issue, and maintaining the config has been really easy because of Lua. Maintaining my own neovim config before moving to Lazyvim has made me learn it to a decent extent, which makes this much easier.

Extras

This is something i just pulled out of reddit recently and would love to share it. Thanks to u/77ilham77, there is a way to move windows without the title bar and I absolutely love it. It's just a simple command and viola you are read. This allows you to hold down <C-CMD> and lets u move the window with your mouse.

defaults write -g NSWindowShouldDragOnGesture -bool true

xkcd comic 196 - command line fu

Meerkat build log

Aditya Hegde — Mon, 22 Apr 2024 00:00:00 +0000

Meerkat is a p2p distributed application to maintain data synchronisation between all nodes using GRPC. Initially part of a hackathon idea proposed (available in this post)

Commits

3f8b029 -> Migrating to GRPC and testing out basic node pinging.
2291e07 -> Implement a basic pool joining protocol.
e9da45e -> Data transmission with failure in edge cases. Using 2 go routines to handle different aspects of the program.
7a51506 -> Creating mutex locks to fix data race issues persistent in the program.
6acc241 -> Basic graceful exit for simple fault tolerance during signal interrupts
b8ffb9f -> Creating Disk Snapshots to identify events of DELETION, ADDITION (updates and addition are broadcasted as the same type of event for now).

Issues

Issue #7: Streaming file contents as chunks
- Current method passes all bytes in one go through a stream. This is an issue with larger files
Issue #6: fix: file deletion broadcasting
- file deletions
- folder operations are shadowed
Issue #5: _ Handle unexpected disconnects from nodes_
- Handle interrupts
- Handle fatal erros
Issue #4: Dealing with merging 2 network pools
- Focus on this issue is to work with network partitions.
- Current system only deals with how a node can enter an existing network pool.
Issue #3: Move towards a gossip protocol
- Broadcast of changes happes to all nodes
- Move to gossip to reduce network overload, the goal is that data propagates to all nodes eventually.
Issue #2: Handshakes and Disconnects not going through
- Duplicate addition of nodes in the client list
- Safe node leaving boradcast doesn't propagate across the nodes
Issue #1: Attempt to connect to client twice

Research for future improvements

Raft Consensus Algorithm Paper
Distributed Systems lecture by Martin Kleppmann

Taking a few messy leaps in a hackathon

Aditya Hegde — Sun, 21 Apr 2024 00:00:00 +0000

Initial Problem Statement

The proposed idea for the hackathon was way different compared to what our final product was supposed to be. What was planned was a live database collaboration implementation where one host, who has the data, allows a few collaborators who all connect together and modify the data. They would communicate through a centralized log server and maintain data correctness.

The approach taken to handle a request coming in, would be single threaded. The host node would work on two separate threads. One to process transaction in memory, and persist changes to the disk. The other thread would be how the host would contribute to the data. All the clients would simply send its operations to the log server (so would the host), and its action would be executed by the host. So this collaboration model followed more of a client-server communication where the host could also contribute to modification of data.

The host would deal with the incoming requests through a TCP listener. This is similar to IO multiplexing. Host handles multiple TCP connections concurrently. It then executes these operations sequentially, ensuring data correctness which would need a different approach using locks when the process is multi threaded.

Creating a a simple echo server

Just to test this connection mechanism, we tried out communication with all the connected peers using a simple echo server. A simple tcp listener was set up in go and all our nodes tapped into this connection.

For now, an array in the host would hold the connection addresses of all the connections to the listener

var connectionChan = []net.Conn{}

Here is a basic sketch of the listener

func TCPActiveListener() {
  tcp, err := net.Listen("tcp", ":8080")

  for {
    // a go routine to accept multiple connections to the
    // tcp server
    go func() {

      conn, err := tcp.Accept()
      connectionChan = append(connectionChan, conn)

      for {
        connResp, err := readBlock(&conn)

        if err == io.EOF {
          log.Printf("[%s] DISCONNECT {EOF}", conn.RemoteAddr())
        }

        if err != nil {
          conn.Close()
          // remove connection from client list
          log.Printf("[%s] DISCONNECT", conn.RemoteAddr())
          break
        }

        echoResponse(message string)
      }
    }()
  }
}

The response is echoed to all connections part of the client list. Building onto this, what was now needed, was to share some kind of instruction from a node to the listener. Decode this instruction and keep the operation in queue for execution.

This initial idea, had a lot of bottle necks.

If the data doesn't reach a log server. The data would be lost. There would be no reflection to a local contribution for a node in the connection.
Data being sent to a centralized server and back to the primary host with the data would be a slow process. Having all commands executed move through this stream would be inefficient

Changes made to the problem statement

On advice of our seniors, we changed the basis of the project. Instead of having a centralized log server handling all the "operation logs" that were to be forwarded to the host, we shifted it to a peer to peer connection where all the data would be distributed and remain in memory. Every local contribution should reflect in the final synced up data across all the node sin this connection pool.

Proof of concept

Creating this decentralized p2p connection, there would need to be a way to figure out how to set up the connection with peers and have the data. There needs to be at least one node who initiates the connection. During this the "host" would act as a server.

Each client in the node would associate a set of data that identifies its existence in the network pool.

┌─────────────┐
│ Client List │ ->  This is a list of clients connected to in the
└─────────────┘     network pool.
┌─────────────┐
│  Local in   │ ->  this is the data that is to be synced up
│   memory    │     when a node in a network has been changed
│    data     │     data persisted to memory is not the priority
└─────────────┘
┌─────────────┐
│   Lamport   │ ->  This is the logical clock associated to the
│    Clock    │     the node in the network
└─────────────┘
┌─────────────┐
│   Command   │ ->  Commands executed by that node. Could be used
│    Head     │     to trace back and help in rebuilding, incase
└─────────────┘     discrepancy during data syncing.

Each client in the connection must maintain such data for validity of connection and also to assist the node to communicate with other connections.

Initial implementation of the communication was built with a TCP server. All data needs to be sent through this. Sending raw instructions cant be possible. Hence a payload was designed assist the receiving node. It would contain appropriate data for processing and also assist the node in what kind of process it would need to execute

┌─────────────┐
│   Header    │ ->  An integer representing the operation to be performed
└─────────────┘     • 0 ->  handle a new connection into network pool
                    • 1 ->  handle a new connection into a pool,
                            (this is when a client is part of a pool
                             and is connecting with the client part of
                             the client list it received)
                    • 2 ->  handle any sort of operation command
┌─────────────┐
│   Lamport   │ ->  The value of lamport clock of the node
│  Clock val  │
└─────────────┘
┌─────────────┐
│   Command   │ ->  Nodes command head at the time
│    Head     │
└─────────────┘
┌─────────────┐
│    Data     │ ->  Data sent over when trying to bring a node to the current
└─────────────┘     synced up version. Primarily when a node enters a pool.
┌─────────────┐
│ Client List │
└─────────────┘

Not all these fields need to be filled up. Only the necessary ones would be filled up and sent over through the connection.

Before sending any data over the connection, we would need to send it over in a way where data can be understandable on the other end. For this, data was marshaled to bytes using the json methods provided go's standard library encoding/json. On the receiving end it would be decoded and would follow the instructions.

Initilizing a network pool

When two clients want to start a p2p connection, for initialization, either one of them would start the connection as a server. The client would then connect to this and send over a payload asking for the nodes client list and data. You would imagine it as two individuals exchanging each others contacts. Along with this the data would get shared over as well. This forms a p2p connection with 2 nodes.

Suppose there exists a network pool and another peer is trying to enter this. As there would be no leader in this connection, they should be able to connect using either one of the nodes. The procedure would follow the same steps as before with an additional one

The entering node requests for the client list of its peers and the data. We assume that in a network, all the nodes have a synced up data. With this in mind the data is sent over to the node entering the network. They add each other to their client lists.

The node goes through the client list and attempts to connect to each one of its clients. It shares it client list, which would be the latest one, and informs them to add them to their client list, hence a connection is formed with each of the remaining clients in the pool. Again an assumption would be all the data would be in sync, hence there wouldn't be a need of sharing that part. In case of any inconsistency, we would rely on reconstructing the data during any other operation communication.

Handling data

The proposed concept was that we would be implementing a basic model of CRDT to make sure all the context of the data is retained when there could be concurrent executions.

With the lamport clocks and the head commands, we could identify if any version of data was behind, and handle the cases accordingly. For example, if there is some sort of mis match regarding the version, the receiver would send its set of instructions to the sender and try brining both versions up to date.

Big roadblock

Like said before, we didn't test out any of the connections. After finishing the first iteration of the client connection system. While testing this, we realized while the data was getting serialized correctly locally, the data seemed to disappear after the receiver node tried unlashing these bytes. We weren't able to get 2 nodes in consensus.

Ignoring the nil value in the image, which was later fixed where instead of adding the connection object to the client list, we were only appending the value of the connection remote address as a string. Even this didn't go through.

Mistakes

We made quite a few mistakes while working with this project. To list out - using TCP as our protocol, using json for serialization of the data. Throughout this hackathon, everything implemented was from scratch. While having premed modules available, we hesitated from using them.

Considering this hackathon was 24 hours long, making everything from scratch set us back compared to our competition. Our initial problem statement being complex, made us work on the theory of the implementation alone for about 4 hours. While the hack began at 5pm, from 8-11pm, we were still unsure about what we wanted to achieve, and where this could be applied.

Only after a few reviews with our seniors, we then began working on the first implementation of this model.

Another big mistake that we made was writing the code for the network connections without testing a single line. While our concept was robust and no apparent roadblock was visible, we didn't test crucial parts of our code.

Our current model of how the nodes communicate with each other and rely on logical clocks would immediately fail, when there is any sort of network partition that occurs.

Learnings

This is something that I'm really eager to work on. Of course, it was a long shot for a 24-hour hackathon. The plan that I put forward was messy to begin with but it was leading us to something. Looking back at it after a day of sleep, it seemed rather foolish that we were trying to reinvent this system. Again we tried doing this all in a few hours.

Thanks to the help of all the seniors, the've been helpful in showing new ways to approach this idea. Our goal set by our senior was to implement this basic distributed p2p connection with synced up data and implement a small text editor to show that such a connection is indeed possible. We weren't able to finish the first part.

It's such a beautiful thing to see systems-oriented projects back at hackathons: (all 1st year students!) one team is attempting to add primitive gc to C, one team is attempting to implement a p2p collaborative text editor... This kinda stuff puts a smile on my face
— Anirudh Rowjee @ override.bsky.social (@AnirudhRowjee) April 19, 2024

A few kind words from my senior Anirudh Rowjee :)

Conclusion

Keeping all the points of failure in mind and also some other technologies and theories suggested by our seniors, I'm planning to take another shot at this. This live data sharing architecture has been bugging my mind for the past few months. Especially since i've been working on with my friend on our project, where we used the Zed, a text editor written in rust, which uses CRDT as a method of maintaining data correctness and allowing live sharing to take place. Ever since, i've been more intrigued to work on something.

The mentors and judges i've spoken to have all been of great help. Just to be a part of the top 10 of this hackathon, definitely made me happy that with such a project theory, we were able to make it recognisable to some people. The mentors have backed us up the hackathon, providing us with great suggestions, and i'm gonna keep these and move forward, work more on this theory. I definitely feel there is something that is possible with it and cannot leave it as another un-finished project.

Link Stash

Aditya Hegde — Fri, 19 Apr 2024 00:00:00 +0000

Here is a collection of links and articles that I found interesting. Also a mini book marking place for myself to remember trying out tools.

homebrew formulas 🍺

mailsy

brew install mailsy
mailsy g

Generates an email. We can use this for mailing lists or something. And access the emails sent over to the email with mailsy m and opens the email in a local webserver.

taskell

brew install taskell 
brew info taskell

wifi-password
mas : mac app store

Notes from Goland Bangalore 75

Aditya Hegde — Sat, 13 Apr 2024 00:00:00 +0000

This post contains a few notes from the Go Blore meetup on 13th April 2024. The speakers covered a few points on profiling in go, hands on demonstrations on it, dependency injection and a talk on how to use go with android.

Profiling with Go

by Swapnil Nakade

Points covered

Using net/http/pprof with go to profile either the CPU, memory or block profiling
Visualisation of profiling using either the CLI, web flow diagram or flame graph
Breaking down the code with profiling and finding places to improve performance

Implementing Zettelkasten in Anna

Aditya Hegde — Wed, 10 Apr 2024 00:00:00 +0000

Proof of concept

Andy Matuschak's working notes is the key inspiration for this concept. We are trying to deviate from the "general idea" of a blog site and focus more on this niche use case. By integrating this feature, we are letting users to create a space to store there "zettels" and share their short notes as well.

We aren't trying to re-invent the process of making an editor that helps users maintain these zettels as there are already some fantastic applications, namely Obsidian, Ginko Writer and Evergreen Notes. Our application as a rather needs to provide a generator to stitch these notes together to make it accessible on the site.

Stages of Generations of Notes

1.0 Figuring out the named links.

All notes usually have titles as a phrase that can be referred to in a certain note. Our job as the SSG is to link these two notes together. For example

Sed ut velit ante. Suspendisse ac porta urna, eget iaculis dui. Lorem ipsum
dolor sit amet, consectetur adipiscing elit. Donec vel enim dolor.
[[Nunc ullamcorper]] neque ut mattis commodo. Morbi bibendum sem accumsan mi
imperdiet, id egestas nulla posuere. Morbi sodales justo euismod nulla
porttitor lobortis sed ut sem.

The above md file is referencing a note namely Nunc ullamcorper. What needs to be done is, this "callout" is to be replaced by a link to that specific "note".

1.1 Basic Working Model

The parser must search through the Body section of these notes. There are supposed to be "user defined" references to notes, which the parser must identify and add. The specific reference to the template data of that specific post is appended with the information of all the links that it has found during parsing of the file. This can be utilised later by the templating engine.

1.2 Automation of linking process

The previous method suggests the user has to manually link posts. With automation, the goal is to remove the need for manually entering these links. Instead we plan to use the [[]] callouts to the note name. For example, [[Nunc ullamcorper]] will reference the markdown file which contains "Nunc ullamcoper" as the Title. These callouts to other notes are to be picked out by the parser and replaced with a proper markdown reference in the buffer, so that the acutal file remains untouched. For example [[Nunc ullamcorper]] will be updated to [Nunc ullamcorper](/notes/zettel_name/123782734234).

1.3 Automation of file creation

This is a step forward from from Automation of linking process. As we are not a text editing application, this feature will make the process of creating subnotes simpler.

Other than just the parser identifying the [[]] callouts, during the live reload process, an additional file will be created under the same zettel, provided a name is mentioned in the callout.

2.0 Restructuring

As of now, our content directory looks somewhat like this:

|— pages markdown files
|— /posts : post dir containing markdown files for all posts

To this, we plan to add an extra directory named notes that will handle all of our zettles. Each zettel (related notes) can be organised in its own sub directories.

Reference:

Zettelkasten creation

Evergreen Notes should be Atomic

We expect users users to specify the head of these zettels by themseves in the frontmatter explicitly

---
title: Note taking can be fun
date: 2024-04-08
type: zettel
head: true
---

2.1 Concept of the `Mega Struct` & Deep Data Merge

As each zettel must have access to the information of all other zettels, the implementation of a Deep Data Merge is quite necessary. Each page is rendered by passsing a Mega Struct that the entire data of the notes section. This struct will have the following fields:

type NotesMerged struct {
    //Stores all the notes
    Notes map[template.URL]Note

    //Stores the links of each note to other notes
    LinkStore map[string][]*NoteStruct
}

LinkStore is a map which contains a slice of pointers to the linked notes which eliminates data redundancy to certain extent. This is an essential feature as Zettel emphasises on dense linking of notes.

This LinkStore Map is the second step of generation after all the notes in the notes directory have been successfully parsed. Once the link maps have been generated, we use a similar render note function to produce the linked html files.

Each Note can is a struct that stores all of the data of a particular note, including the frontmatter.

type Note struct {
    // Note data including frontmatter and content
    LinkedNotes   []string
}

TODO for zettelkasten impl

Generation of Linked Notes
- Implement 1.1 version of linking (user defined references to notes)
- Implement automation for the process of linking. Using [[]] callouts to file names.
Tests:
- unit tests for parsing parocess of the package rendering processes of package

Presenting anna at fireside

Aditya Hegde — Thu, 04 Apr 2024 00:00:00 +0000

There are several amazing SSGs out there, like Hugo and 11ty. Building your own SSG is an amazing learning experience. It also motivates one to maintain and improve their personal site.

Presented and written by Adhesh, Anirudh, Aditya and Nathan

Building personal blogs from the ground up can be a tedious process. Some of us have had our hands deep in vanilla HTML and CSS, which isn't fun to maintain. We all want to get to the point and share our thoughts on the web. But, there is a small bump that stops us from doing so.

Maintaining your personal site is like working with your own Neovim configuration. Every issue fixed would lead to an entirely unrelated bug. There is a lot of time spent fixing things rather than getting productive work done.

A static site generator is an immensely useful application

It can simplify the whole process: allowing you to spend time and energy on quality content. Keep reading to find out how we designed anna v1.0.0

Introduction

ACM-PESU ECC conducts the ACM Industrial Experience Program (AIEP), an annual program that spans six weeks.

It involves students working as a team to develop an industrial-level project. AIEP intends to give students hands-on experience with real-world projects. It is an excellent opportunity to interact with like-minded individuals.

Our AIEP team consisted of Adhesh, Aditya, Nathan, and Anirudh.

Our mentors (cool ass senior names!) gave us some great ideas for a team of us four freshers. We were puzzled whether to build a distributed Postgres clone or a load balancer.

Deep discussions over a week got us to the topic of making blog sites and how tiring it is to work with, which only gets worse as you write more and more content for your internet home.

This and inspirations from Saaru and Sapling pushed us to tackle this problem with our own SSG.

    ___
   /   |  ____  ____  ____ _
  / /| | / __ \/ __ \/ __ `/
 / ___ |/ / / / / / / /_/ /
/_/  |_/_/ /_/_/ /_/\__,_/

A static site generator in Go

The small but big decision!

Anna is written in Go. We considered writing it in Rust, but that came with a steep learning curve. Go is a powerful language and has excellent concurrency support, which suited our requirements to build a performant application.

What's in a name?

Probably the first thing that us four came across when joining ACM and HSP was the famous Saaru repository. Saaru, one of the projects that inspired our ssg, is derived from a Kannada word. Saaru is a thin lentil soup, usually served with rice.

In Kannada, rice is referred to as 'anna'(ಅನ್ನ) pronounced /ɐnːɐ/

This was just a playful stunt that we played. We plan on beating Saaru at build times, optimizing at runtime.

Genesis

We began the project in a unique manner, with each of us creating our own prototype of the SSG. This was done to familiarize everyone with the Go toolchain.

The first version of the SSG did just three things. It rendered markdown (stored in files in a content folder in the project directory) to HTML. This HTML was injected into a layout.html file and served over a local web server. Later, we implemented a front matter YAML parser to retrieve page metadata

What made us develop this to a great extent?

Beginner-friendly: An easy setup wizard, easy and ready to use layouts, and themes. We want the process of typing out a blog and putting it up on your site to be short and simple.
Speed: Be fast (hugo – written in Go, is remarkably fast)
Maintainable: This ssg will be used by us, hence it should be easy to fix bugs and add new features
Learning curve: None of us have really shipped a production ready application. Since AIEP is all about making industry-ready projects, we chose to use go: so we could spend more writing code and not worrying about our toolchain or escaping dependency hell.
Owning a piece of the internet: Aditya and Anirudh registered their own domain names. Now their anna sites live on [hegde.live] and [sudhir.live]

Benchmarks! Can anna lift??

In simple terms, to beat Saaru's render time (P.S. we did!). Something you probably never notice while deploying, but it is what pushed us to spend hours on this.

Adhesh was pretty excited to pick up Go and implement profiling, shaving milliseconds off-of build times, when he implemented parallel rendering using goroutines.

Prototype

The initial prototype built by Adhesh consisted of a multi-goroutine system. A new goroutine would be spawned to walk the required directories. If the current path being walked was a file, the path would be passed to another function along with its current modification time.

The previous mod time of the file would then be retrieved from a map holding the mod times of all the files:

If the given file was freshly created, its modification time would be added to the map.
If there was no change in the mod time, no changes would be made.
If there was a change between the current and previous mod times, another function would be called.

The new function checks if a child process is running:

For the first render, when a process has not been created, a new process is created that runs anna ("go run main.go --serve")
For successive renders, the existing process is killed and a new process is spawned once again that runs anna.

This prototype was not very efficient as it created and killed processes for every change. It had multiple goroutines attempting to walk the directories at the same time. It also used multiple mutual exclusion locks to prevent data races. Integrating this into the project also proved to be challenging.

Improved version

The live reload feature was improved by Anirudh. The updated version utilised two goroutines.

The main goroutine used the earlier file walker, with one important change: it sequentially traversed the directory without spawning new goroutines. For any modification to a file in the current traversal, a vanilla render of the entire site would be performed. The goroutine would then sleep for a specified duration (currently 1 second) before attempting the next directory traversal.

The secondary goroutine ran a local web server that served the rendered/ directory.

This eliminated all locks and avoided the creation and destruction of any child processes.

We cook! 🍳

Here are some screenshots out of our group chats, that demonstrate build times, profiling et-al when having thousands of markdown files or in this case just copy-pasting a single markdown file en-mass!

After about 2 weeks of training (ahem) coding, we had a (merge) bringing parallel rendering and profiling to the table

Profiling

Heres the CPU profile of anna, generated using pprof. This profile was generated while rendering this site.

Here's an SVG showing how much time each sys-call / process takes and how each block adds-up to render / build times

You may wanna zoom-in about 3-4x times to get to see how our ssg works

A big rewrite (when we went for a TDD approach)

Starting off this project, we kept adding functionality without optimization. We didn’t have a proper structure; PRs would keep breaking features and overwriting functions written by fellow team-mates.

A new proposed rendering system

We proceeded to restructure our SSG into: modules previously part of cmd/anna/utils.go and cmd/anna/main.go were to be split between pkg/parsers/, pkg/engine/ and pkg/helper

pkg
├─── helpers
│   ├─── helpers.go
│   └─── helper_test.go
├─── engine
│   ├─── anna_engine.go
│   ├─── anna_engine_test.go
│   ├─── engine.go
│   ├─── engine_test.go
│   ├─── user_engine.go
│   ├─── user_engine_test.go
│   └─── engine_integration_test.go
└─── parsers
	├── parser.go
	├── parser_test.go
	└── parser_integration_test.go

Currently there are two separate types of files that have to be rendered. One set includes user-defined files such as index.md, docs.md and various posts. These are specific to a user.

The second set of files that are rendered include tags.html, sub-tags.html and posts.html Now, the generator/engine has a method to render "anna specific" pages and another method to render "user defined" pages which include all the user pages and posts

Here's some of Anirudh and Hegde's work written during week-2

Refactored main.go to only handle flags
Wrote unit and integration tests for the parser and engine package
Split the rendering system to make parallelisation easier by switching to a three method system.
Render "user defined" pages which include all markdown files and posts (This method has been parallelised)
Render tags and tag-sub pages separately, which could be parallelised in the future
Wrote a benchmark for main.go that times the entire application

To search or not to search? 🤔

That is the question > Is our static site becoming and at what cost?

We were wondering if we’d need a search function on our site since Google and any other web-crawler index our site anyway. If we needed to implement it, we had a constraint: we cannot use an API. It had to be static and local to be user-friendly to work with. Aditya and Anirudh implemented a JSON index generator that uses "Deep Data Merge" to index posts on our site.

This index is built at runtime and works without any lag or noticeable delay when searching across posts. We mean to re-write it using WASM if necessary and if it costs us time when performing searches.

JS integration as plugins

Aditya added a field to our frontmatter which lets you pick and add certain JS based snippets to your site. This way, you get to add highlight.js support, analytics scripts and donation page widgets; that you can source from the static/scripts folder and toggle as needed per-markdown page.

Wizard

Nathan proceeded to work on a GUI; a web-based wizard that let's a new user setup anna along with a couple of easter eggs along the way 🍚

The wizard lets a user pick a theme, enter your name, pick navbar elements, and validates fields using regex checks so you don’t need to worry about relative paths in baseURLs, canonical links, and sitemaps. After successfully completing the setup, the wizard launches a live preview of your site in a new tab.

Raw HTML

What if you'd want to add a contact form to your site? or embed YouTube videos or iframes of your choosing?

Anna let's us do that! Although, the point of a static site generator is to quickly get to writing and focusing on the content. You can still embed js elements and iframe as needed to showcase any interesting YouTube videos or to just rickroll people!

Conclusion

We are at week: 3/6 and have a lot of things in store and bugs to squash!

Today anna is tagged at v1.0.0 and we use it on our personal sites: hegde.live // sudhir.live // polarhive.net

Quality of work

Aditya Hegde — Sat, 09 Mar 2024 00:00:00 +0000

Parallax in approach

A few thoughts off of my mind, my perspective on effort. It's something I'm a bit particular about. For me, there is a wall separating work achieved from hard work and work done for the sake of it. A wall that no one can pass through unless you change your approach or perspective of a situation.

I'm a person who is particular about my job. I tend to put myself down and set unrealistic expectations at times. Now people would say, I'm crazy for it. But, it's that mindset for me that makes me want to do more.

Every second part of your effort on something counts, it makes a difference. For me, the end result must showcase that effort. You can't spend hours together working on something, only for the end result to be garbage. If it doesn't go in hand with the effort that pooled in.

Navigating perspectives

Being true to your work is one more thing that is very important. Set your margins high, as not everyone has the same perspective. Don't ever make assumptions about whether the other party is on the same wavelength as you are. Things break down real quick the moment that happens. You never expect it and then it hits you, had in the back. All that effort, gone down the drain, and now you're at square one.

xkcd comic 1658 - estimating time

Snapshots

Aditya Hegde — Sat, 24 Feb 2024 00:00:00 +0000

Camera gear: Cannon EOS Rebel XSi (no idea about the lens) and Sony a6700 with a Sony E PZ G 18-105 mm F4 G Lens

Click on any image for the high resolution version. You can find my other stills on unsplash

Aditya Hegde

Aditya Hegde — Sat, 24 Feb 2024 00:00:00 +0000

Aditya Hegde

Hello world! I'm Aditya Hegde. Currently interning at Akamai. Prev. research intern at PES Innovation Lab (prev. Microsoft Innovation Lab).

Currently pursuing my bachelors degree in CSE at PESU. I have a fond interest in building systems and storage. Outside of my work, I spend my time trying to play the Guitar, am a "hobby photographer" and love rock and coffee.

I'm part of a small community called HSP in my university and together run a the FOSS team homebrew where we try to grow and network beyond our university in different meetups and workshops. Since my internship at PIL (PES Innovation Lab), I am part of the research lab as a member where we maintain the infra for events and mentor the incomming intern batches.

This site came about after my frustrations of maintaining a simple webpage without moving to a JS framework. A couple of my friends and I built anna, which is what is used to build this site.

This site has a bunch of semi-organised posts that I try writing once in a while and TIL's (huge inspiration from Simon Willson's TIL page at til.simonwillison.net), where I will be dumping links and other side notes that I take while learning and when I attend talks.

Connect with me on Github, Linkedin, Unsplash, 𝕏 or mail me at mail@adihegde.com

The design on this site derives its inspiration from these folks: eatonphil.com, matklad.github.io,polarhive.net, rowjee.com, icyphox.sh

par moi

Aditya Hegde — Sat, 24 Feb 2024 00:00:00 +0000

My first HSP talk

Aditya Hegde — Sat, 09 Dec 2023 00:00:00 +0000

I recently got the opportunity to take part in the project expo hosted by the club HSP part of PESU ECity campus. As part of this expo, I presented one of my on going projects xkcd-grab in-front of an amazing crowd. So lets get to the actual part of the blog, what this presentation was all about.

This relatively small project is somewhat of a playground for me to explore different searching algorithms and querying techniques. While this might have a niche target, I want to build this tool into a more robust API client. The roadmap of this current is to make a smart cli tool to find the most relevant comic based on a search query. The CLI itself is a minimal working tool I wrote with python. Besides using the api, i've created several extra features that completely changed original idea of the app.

This project begun in 2021, during my college entrance preparations. It started off as a basic API tool that worked with the xkcd api to fetch a comic based on a derised number. Looked cool at the time but was still pretty clunky. Why would you want to search something based on a number. Thats horrible. Well, the cli tool couldn't do that initially. This is when decided to implement some cool way you can interact with the cli, and find the comic right from your terminal.

Fuzzy finding with levenshtine distance

Once i've got my hands on all the data needed, now was the time that I find a way to give the user the queried comic. My goal was to retrieve a comic based on a string, probably a mini description and use vector searching to find the most relevant comic. Seems good on paper, problem was the data. The api didn't provide any sort of description for over 1000+ comics. And with the limited time of 3 days, along with hours of college work, this approach wasn't feasible. The only way I could scrape out data would be using and OCR. The images of these comics have far better insight about whats happening thatn the description...mostly because there was nothing available. I will bring this up in a bit while talking about the little easter egg feature.

Now the only other option would be to use the titles of the comic. What the task was now is to achieve the best string approximation and provide the user with the 10 best matches. Levenshtein distance out of all the options was the simplest to implement given the time. Thing is I couldn't get this to work with an implementation that I craeted from defnition of the method to determine the levenshtine ratio. While getting the distance was a simple formula. Finding the ratio of kept me stuck for a while. It worked just fine, but the results didn't seem to align with premade modules that are supposed to do the same thing. While my results for the ratio were somewhat right, they were off by around 0.2 to 0.3. Turns out, the way fuzzywuzzy calculates the ratio is by using a slightly modified version of this formula.

By definition, if l is the Levenshtein distance, and m is the length of the longest of the two words, the ratio is given by 1-(l/m). However in the modified version, the sum of the lengths of the words were used for getting the value.

def Levi_ratio(a: str, b: str) -> int:
    d = levi_distance(a, b)
    l = len(a) + len(b)
    return 1 - (d/l)

This was womething that I found out hours before the final presentation. So I had to go ahead with using the prebuilt module, which im not so proud. The results were good but not fully upto my expectation.

Here is a small demo of all of these things together. I appologise for the shitty quality of the gif. I yanked them from the presentation, and google slides doesn't like you downloading concents from a slide :P.

Problems encountered

This project would have been a smooth sail if it werent for the big obstacle that I encountered where the data was simply insufficient. As a solution to this, I created a simple web scraper that works neatly along with the pre existing system that I made for the fuzzy finder. Instead of iteracting with the xkcd api, a neat google search querry along with some web scrappign gave some of the best output...I mean what else would I expect. Each xkcd url is in the following format.

https://xkcd.com/<some number>/

Here is the basic structure of a Comic class.

class Comic:
    self.num        # comic number
    self.url        # comci url (not the api endpoint)
    self.title      # comic title
    self.trans      # comic transcript (missing for over 1000+ comics)
    self.alt        # comic alt text
    self.img        # comic image url
    self.content    # comic content

The content is stripped out from the images using googles tessaract OCR engine, which has been made available to python.

How this app works is it dumps all the data locally in a csv for quick access in effort to use the api as less as possible. Ofcourse there a ton of comics that need to be retrived. In addition to the api call, running a sequential task of using an OCR to get the data from the comics is way too time consuming, especially when you are handling with anything over 1000s in number. Hence there was a need for a simple parallelisation operation

def get_session():
    if not hasattr(thread_local, "session"):
        thread_local.session = requests.Session()
    return thread_local.session

# Returns a list of data in the same order as the csv file
def get_comic_data(comic_num):
    comic_data = Comic(num=comic_num) # this neatly give me a Comic object that i can handle
    # ...
    print(f"Data for comic {comic_num} fetched sucessfully")
    comic_data_nlist.append(data)


# Threading function to get all comcis
def fetch_all_comics(nums_list):
    with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
        executor.map(get_comic_data, nums_list)

    print("++++++ APPENDING & UPDATING COMICS +++++++")

What was supposed to be a 2 hour job of sequentially handling all the tasks got reduced down to 5 minutes (5 minutes and 23 seconds to be exact :P). This function is automatically called when any operation is called for using the CLI tool. Hence making data available for latest queries as well as for the fuzzy finder.

Hello World

Aditya Hegde — Fri, 08 Dec 2023 00:00:00 +0000

I'm not usually great at these things, but I want to give it a try. So here it goes.

Hey There! My name is Aditya Hegde and this is my small space on such a vast land of the web. What about me? Well, I'm a freshman at PES University. Parts of some cool clubs such as HSP and ACM which was my first little family here in college. Besides the stressful parts of it, just the first 6 months alone have shown me a whole new world.

It's been a whole semester since I've joined college, and I could just go on and on about the number of amazing people I've met. It's like a whole new wave I've never experienced. Been part of one of the coolest mentoring teams, mentored in some amazing workshops, and even had the opportunity to have my presentation on one of my projects.

This site is special to me. Not only it is the place where I would like to share my work but also it is being generated by anna, which is one of the first big projects I've worked on with an amazing team Anirudh, Adhesh and Nathan. As of now, it's still a work in progress, and working on it has been great.

failure doesn't define a person

I love learning new stuff. And at the same time, I don't know what to focus on. So as of now, I'd be lying if I said I'm specifically into some system-level stuff or networking concepts... Whatever it is I'd like to share it, and what better place to do so other than my page. I believe failure does make someone stronger. There are so many physicists out there, great ones who tried but failed to read some conclusions, but ended up with something new, sometimes something revolutionary. There's so much more to learn.

404

Aditya Hegde — Thu, 01 Jan 1970 00:00:00 +0000

Woops, you seem to be lost :)

GoBlr #86 30 May 2026

Go routine and channel internals

Rust Bangalore Meetup 16 May 2026

Platform Engineering Meetup 9 May 2026

Bengaluru Systems Meetup 25 April 2026

Boehm’s curve in life

You can not have the cake and eat it too

Lost a bit of spark

Things to remember

Bengaluru Systems 11

Overview

My talk on Quaso

Images

Presentation

Twitter Post

Pes Innovation Lab

I purchased a new camera

Studying db hexdumps

Building systems in rust

A much needed Introduction to this Blog

How this project started out

SEP 23

SEP 24

SEP 30

Back at it again...changing everything

21/12/2024 Almost There

Reason for the previous undefined behaviour

Up next

24/12/2024 1:30am

24/12/2024 8:20pm

24/12/2024 11:07pm

04/01/2025

08/01/2025

Wrapping Up

References

Goland Bangalore 76

Lunch with a Pigeon

Techstack

Text Editor

Window Manager

Terminal Emulator

Extras

Meerkat build log

Commits

Issues

Research for future improvements

Taking a few messy leaps in a hackathon

Initial Problem Statement

Creating a a simple echo server

Changes made to the problem statement

Proof of concept

Initilizing a network pool

Handling data

Big roadblock

Mistakes

Learnings

Conclusion

Link Stash

homebrew formulas 🍺

Notes from Goland Bangalore 75

Profiling with Go

Points covered

Implementing Zettelkasten in Anna

Proof of concept

Stages of Generations of Notes

1.0 Figuring out the named links.

1.1 Basic Working Model

1.2 Automation of linking process

1.3 Automation of file creation

2.0 Restructuring

2.1 Concept of the Mega Struct & Deep Data Merge

TODO for zettelkasten impl

Presenting anna at fireside

Introduction

The small but big decision!

What's in a name?

Genesis

What made us develop this to a great extent?

Benchmarks! Can anna lift??

Prototype

2.1 Concept of the `Mega Struct` & Deep Data Merge