Whenever you’re working on something, it’s often a good idea to research best practices and how you’d implement them. The easiest way to avoid falling into tons of technical debt, even if it’s just a hobby project, is to start with a good idea of what your final architecture will look like. But sometimes you want to do something that doesn’t really have detailed examples; you don’t have someone telling you what the best practices look like and how you would implement them.

This blog post is a tiny little rant going over the problem that I wanted to solve and how I eventually arrived at what I consider the “best” way of solving it, with very little guided documentation. I don’t expect others to read this, because it really is just me rambling about a non-issue.

The Problem

Asynchronous programming is difficult. Especially when having to deal with “state”, it is hard to prevent deadlocking and complex logic. In my case, I was creating an MQTT Broker in Rust. I had many clients talking to one centralized server, and the server needed to store information about each client, as well as communicate information between clients. Now maybe for the Rust experts and asynchronous programmers, the solution to something like this seems obvious. Apparently so obvious, that there are essentially 0 guides about designing an asynchronous “architecture” in Rust.

My first intuition was to spawn an asynchronous callback for every single new client and store all their information in the broker itself. Perfect! All my clients were signing on and ping-ponging, but the moment I tried to communicate between them, it all went to shit. In a language like Python or C++, to store the state associated with each user, I would create some sort of hashmap or dictionary, and connect each client’s ID to the information I need to store. These languages would let me continue my journey with no regard for what would happen if I tried to modify this dictionary across processes and I would be happy to do so. Rust was not so kind.

Sharing Memory

Rather than allowing me to throw state around willy-nilly, Rust started complaining about something called “thread safety”. Absurd. Any attempt to access state between the threads would complain that BrokerState cannot be sent between threads safely. After some research into the documentation and the Rust book, I discovered the beautiful prison of Arc<Mutex<T>>. Coming from more permissive languages, I was just looking to get this done in any way possible.

So I replaced my BrokerState with an Arc<Mutex<BrokerState>>, threw some state.lock()s in there and hoped that all was well. Spoiler: it was not.

What I wanted essentially, was for every single client to have ownership of the central state. The Broker just acted as a centralized mediator between clients and their interactions with each other. If this sounds like a bad idea to you, you put more thought into this than I did. I had deadlocks all over the place, syncing errors, and a terrible ownership system. Fixing one issue always led to another. Even when I got the program to compile and run, it deadlocked. Everyone I asked agreed that the way I was doing things should work. But it was painful. And Rust was there at every step, questioning everything I did, making it so much harder to get something that even compiled. It let me keep going, but it wanted me to stop.

Eventually, it got to me. I took a step back and thought about what I was trying to do. I deleted all the code I had written and started from zero.

Channels

For some reason, I approached asynchronous programming in Rust the same way I approach asynchronous programming in Python or Javascript. I probably should’ve known that this would not work.

I dived much deeper. There are surprisingly very few write-ups about an actual, complex, asynchronous architecture in Rust. After this little rant, I want to do a more technical writeup of my MQTT Broker to provide insight to those that are struggling like I was. Instead, I read through different repositories, looking at people’s HTTP servers, their file servers, some weather apps. Some of them were like mine: Arc<Mutex<T>> in every crevice. But then I found something that I had heard in passing, but never really seen applied: message channels. I’ve heard of channels, I’ve toyed with Kafka, but I’ve never really found an actual use case for them in my hobby projects.

In Python, C++, Java, or most languages that aren’t designed with safety in mind, I would never have stopped and reconsidered what I was doing, but Rust made it so painful to keep going that I had to stop. It’s almost a testament to Rust. I could implement this system the “normal” way, but it would be bug-prone and a terrible mess. Rust makes it so, so painful to implement this client-broker architecture the “normal” way, that I needed to think about better ways to implement what I was doing.

The architecture ended up being ridiculously simple and clean. I was in awe of how quickly the complexity of my program decreased. Rather than allowing every client to directly modify the state. I gave each user a channel to communicate with the broker, and the broker could send specific messages to the client. The client would send a message to the broker, kindly asking it to change the state, never actually having to touch it.

The Language Itself

Had I been doing this in some other language, maybe I would’ve encountered more resources about proper architecture, but the language would have let me share memory and be unsafe without any repercussions until it was too late. There are a lot of ways to discover the best practices when approaching a new task, and often there is a lot of documentation and resources about it. There definitely is documentation for Rust and Tokio (Rust’s “major” asynchronous programming library), but it was convoluted and not really built around real-life examples.

What ended up happening is that the language itself yelled at me when I tried to do something poorly. It never pointed me in the right direction, but it told me I was going the wrong way. So, with a lack of tutorials and examples, I dived into codebases.

Finding Best Practices

There were really two things that led me to “discover” what I now consider to be the best architecture for a project like mine when there were no documented “best practices”. Firstly, the Rust compiler’s insistence on keeping me thread-safe and preventing bugs led me to realize that I was doing this very incorrectly. Secondly, just diving into other codebases.

I realized that I often don’t consider how ideas transfer between projects. There are not very many MQTT brokers out there, and even fewer written using modern Rust and Tokio, so I had no luck looking at actual examples of exactly what I was doing. When writing something more “common”, like an emulator or a compiler or even a website, I often look at other examples and see how they handle certain core tasks. For example, there are a lot of different ways to handle opcode lookup in an emulator, and a lot of those methods are terrible and will lead to awful looking code. The easiest way to avoid one of those bad methods is to just see how other people do it. What I never considered was that other projects, even if they were doing something very different from mine, could still be used to extract ideas from.

So that’s what I did. I looked at weather apps, HTTP servers, anything using Tokio that I could find. I reproduced the architecture that these popular projects were using and realized I could apply it too. With this better understanding of Rust and Tokio, I intend to do a write-up of my project and help others that might be struggling with using Tokio.

As I learn more about Rust and Tokio and become familiar with the environments, I realize that I didn’t make use of the documentation that was available to me. The problem with a lot of the documentation though, is that I couldn’t really use it. I understood message passing with foos and bars, but when it came to actual examples, I was out of luck. Maybe it’s the way I learn, but I feel like most people need better, concrete examples to discover best practices and make use of something.