Always choose simple

Today I was tasked with something rather trivial at work, and I wrote the simplest, dumbest code possible to solve the problem. A colleague of mine was surprised looking at my patch, she said it's so simple and beautiful, as she expected some more involved cleanup or refactoring. This brought to mind a little story I'd like to share.

When I was 20 I got my first job at a startup. The product we were working on could now be described as a CMS (“Content Management System”), but it was quite advanced and ahead of its time. Those were days when Netscape Navigator was still a thing and if you made a website, it better work well on that piece of crap browser. I used to envy people with Windows, because Internet Explorer 4 was clearly better than Netscape.

Our CMS was a full blown product, with the client (user interface) written in Java. Java stood to its promise — write once, run anywhere. But you better run it on Windows, because on Linux the Java experience was utterly miserable — slow and horribly ugly. But I digress. Our CMS also had its own in-house HTTP server, database abstraction layer, PDF generation library (I wrote that), and on and on. Tons of components. As a C++ engineer, I worked on the backend. Junior as I was, I knew C++ for years and considered myself pretty good. I took the interview with flying colors. I knew all those arcane details like static_cast vs dynamic_cast, templates, memory etc. heck, I was even comfortable to write assembly code. Yet there was this guy, Rykov (Alexander, I think?), who was a senior and constantly rated our code like 5/10 in reviews. We kinda hated him.

One day me and a colleague were tasked to figure out why our server is only able to process like 0.7 requests per second, and how can it be improved. Boss even promised bonuses proportional with the improvement percent. After much digging, we did find an improvement so big that our bonuses would have gone through the roof, had he kept his word. Here it is: the main entry point to our server was a single C file, pretty small too. That one was written by Rykov entirely. It just waited for incoming connections. When a request came in, it forked a child process to handle it. The child terminated after each request. By poking around the code, we figured out that if the child *didn't* die after the first request, it served the second one much faster. By *much faster* I mean something like 50 times, if I recall correctly. We were baffled by it. We measured how much time it takes to fork, because we thought that was the bottleneck, but it wasn't. The fork() call itself took exactly 0.0 milliseconds every time. Eventually we figured out that that's how it works. fork() is instant because it doesn't really do anything, it just creates a child process — but when it first runs, that child process will need to duplicate memory that it shared with the parent, to eventually create a full copy — it's copy-on-write. The first run is always slower, and by terminating the child after it served a request we effectively lose the opportunity to serve a subsequent request much faster. All this is common knowledge, but it was new to us then (oh, so that was the reason why Apache had a worker process pool!).

So any case, we reported our findings and over the next few days we started to put together a comprehensive infrastructure for handling a worker process pool, with tons of objects like service factories and monitors and configurators and what not. The C++ design patterns book was still a fresh success at that time. We wrote hundreds of lines of code which wasn't yet working because there were still missing methods and objects and abstractions that we had yet to invent. And then one day we learned that Rykov had already committed a 10 lines patch that does the job just fine.

I looked at his code in disbelief. I forgot the specifics of how it worked, but I remember being amazed at it. It was small, simple and beautiful. And I was pissed that I had to drop my grand architecture on the floor, but I had to admit that his 10 lines patch worked well and there was no reason for all the crap I wanted to come up with.

I don't remember if I ever told Rykov about that feeling. I was too young, too proud, maybe too full of myself. But I learned the lesson. Mr. Rykov, if you're reading this by any chance, I hope you are well! :)

One comment. ONE! Gimme some more!

Add your comment

# anonymous
2022-10-26 20:25
The reason the Rykov dude, who was indeed extremely good, despite being an asshole, was forking for each request because there were memory leaks in the application. My guess is it was one of the few times he was sloppy and couldn't have predicted the crap he was getting into. The solution for fixing was not very scientific, but it was something that came with experience. Instead of forking the process on each request, he was processing 100 (or something) requests (ignoring the memory leaks) and then forking (ignoring one slow request). So in the end it was just a workaround made to fix some overly complex code, using a million patterns that nobody could read.
Oct
21
2022

Always choose simple

  • Published: 2022-10-21
  • Modified: 2022-10-22 10:42
  • By: Mishoo
  • Comments: 1 (add)