A Sixteen-Year Chain of Problems

Alex Harbuzenko · · .Net, csharp, gamedev, server_side_optimization

Not too long ago, at the dawn of this millennium, on a cold November day in 2004, I sat down to write a server emulator for a certain online game. Writing it went quite well, in the pleasant-to-the-eye C# and .Net Framework 1.1. I didn’t set myself any particular goals, and my experience was relatively modest. For some reason the community took a liking to this little project (maybe because it appeared before the official launch of the main game?), and within a few months I was facing explosive online growth, along with serious performance problems. The project lived for 6+ years, reached some notable heights (2500 concurrent users at its peak, around 20000 MAU), and then passed away peacefully. And now, a decade and a half later, I decided to build my own MMO on that same “time-tested” codebase, and ran into the very same problems, despite having already solved them once.

P.S. No IP was harmed in the writing of this article. The original project, although steeped in a spirit of piracy (a free server for a paid game!), didn’t infringe on anyone’s rights: none of the rights holder’s code was used, and the server was built entirely on research into a legitimately purchased game client, plus the developer’s own common sense. This piece only talks about the challenges the author ran into and the original ways of solving them, both in the old project and the current one. I apologize in advance for the narrative style, as opposed to a plain list of facts.

Introduction

You can argue all you want about .Net not being suited for servers, but back then (and still now) it seemed like a pretty sound idea to me: write the logic as scripts, compile them, and load them on the fly, without worrying too much about memory allocation, garbage collection, pointers, and so on. In essence, this lets you delegate business-logic scripting to less experienced developers, with just a code review as the safety net. But for that to work, you first need to be sure the core itself runs without hiccups – and it started misbehaving at just 10-15 concurrent users, both in 2004 and in 2020.

In 2004, everything ran on Windows Server 2003, .Net 1.1, and MSSQL 2000. The server and hosting were provided by the ISP Wnet, and later a new server was built with donations from players. The project was strictly non-commercial, and the small income from banner ads and premium accounts went toward upgrades.

The modern server runs on Mono under Debian, in .Net 4.7 compatibility mode, with MariaDB for data, hosted in the Hetzner cloud. Gone is the wide-eyed idealist who believed games should be free, and that donations and item sales kill all the fun. These days that character has gone considerably grayer, traded enthusiasm for experience, and is convinced a startup should bring both enjoyment and income.

But this isn’t a story about that – it’s about homegrown servers and their problems.

Chapter 1. Pestilence

On my local machine, with a single player, the server always worked flawlessly. On the dedicated machine with live players, though, the server process would sometimes just quit outright, with an odd preference for crashing at night, right when the admins were asleep. It was almost funny: users would call the provider’s support line, the support guys would power-cycle the server, and by morning there was no way to figure out what had caused it. The server ran as a service, so it restarted without any trouble, but the logs showed no obvious cause for the crash. I tried running the process from Visual Studio a few times, but the debugger would detach after a while, and performance suffered badly besides. The system EventLog, meanwhile, turned out to be remarkably uninformative.

The first thing I did was significantly improve the logging system – it started capturing everything written to Console.Out and Console.Error. Then I added an UnhandledExceptionHandler, which also logged error information. The log could run in AutoFlush = true mode, which cost some performance but protected against messages getting cut off mid-sentence.

The next step was converting the service into a console program and launching it from a separate cmd window – that way I could find the window and try to read the last thing the server was trying to tell us. This, incidentally, led to a spectacular problem: the server would stop dead because someone clicked in the console window, triggering text-selection mode and freezing the process. At some point I was writing two logs in parallel – one through .Net, and one via redirecting to >> log.txt.

Unfortunately, UnhandledExceptionHandler doesn’t help with three kinds of exceptions: OutOfMemoryException (any action inside the handler might itself need to allocate memory, and so won’t run), StackOverflowException, and exceptions raised in unmanaged code. And as it turned out, I had at least two of these – an access violation somewhere in native code, and OOM.
To fight the access violations, I gathered my courage and got rid of native code entirely – no more ZLib (replaced with ICSharpCode.SharpZipLib), no more OpenSSL (I wrote my own SRP-6 implementation), no more native MySQL connector (switched to System.Data and MSSQL for the database).

Separately, I discovered that throwing an exception inside a Socket.BeginReceive callback could kill the entire process. That callback runs on the .Net Thread Pool (specifically in the part of the pool reserved for IO threads), and if it throws, it brings down the whole program, UnhandledExceptionHandler notwithstanding. And a bit later it turned out that the BeginReceive->EndReceive->BeginReceive call chain also causes problems and needs to be reworked so BeginReceive isn’t called from inside the callback.
All of this substantially improved the picture, and the server started crashing far less often – mostly only when it ran out of memory.

By 2020, the server application was purely a console program, running inside a separate screen session on Linux. Running it under Visual Studio was no longer an option, but over the years the logger had become quite sophisticated, UnhandledExceptions were rare as hen’s teeth, and there was no native code at all anymore. That said, it didn’t save me from OOM and StackOverflowException crashes. The stack depth in the StackOverflowException case had grown by an order of magnitude, filling hundreds of kilobytes of log with identical repeated messages and refusing to print a proper stack trace. Either way, redirecting to >> log.txt quickly let me figure out who and what was to blame. A Telegram bot that pinged me whenever the server process died also helped a lot.

After that it was just a matter of technique. Digging through the logs showed the stack overflow was happening not in the core, but in the business logic: a missile would collide with another missile or a mine, they’d detonate, that would trigger the detonation of the first missile, and so on in a loop. Ordinarily that’s just a routine bug, but that was the moment I felt a strange sense of déjà vu, fighting long-forgotten demons from the past. And then a new (or long-forgotten old) cause of pestilence appeared – resource starvation.

Chapter 2. Famine

Servers at the turn of the century could pack an unbelievable amount of memory – on my home computer I made do with 256MB of RAM, while the server had a whole gigabyte! And it still wasn’t enough – the memory was slowly but steadily eaten away, leaking like water, and as the player count grew, the countdown to a crash was measured not in days but in hours. Every time I started the server, I’d wait in dread for the resource shortage to trigger an OOM or something even worse. Of course, I wasn’t just sitting on my hands: I repeatedly tried attaching to the running server with Visual Studio (with memory this tight, that turned into torture and then a crash), tried taking a memory dump with the WinDbg tools (no luck), and once even managed to launch the server under dotTrace (briefly). A good stopgap was adding another gigabyte stick of RAM, and later moving the database onto a separate machine. But the problem persisted – the server’s memory usage would slowly but surely creep up toward the 1.7GB ceiling, and then it would crash. Tweaking the page file settings didn’t help. Separately, I started noticing 100% CPU load more and more often. The server was flat-out starving, short on resources, and I couldn’t figure out why – even on my home machine, after several hours of running, it never went above a reasonable ~100 megabytes. The real revelation came from articles by Maoni Stephens and Rico Mariani about the GC, the awkward term LOH (Large Object Heap), and how .Net organizes memory. It looked like every network operation pinned its buffers in memory, which automatically pushed them into Gen 2, and for large enough sizes, straight into the LOH – a special circle of hell reserved for the fattest, biggest objects. I fought them like a young David – writing my own buffer pools, reusing arrays wherever I could, implementing my own collections that pulled arrays from a pool and returned them afterward (keep in mind, this was .Net 1.1, no generics!). But Goliath stood his ground: memory still leaked, just a bit more slowly, and the starvation didn’t let up. So I gathered my strength and started allocating buffers on the native heap via Marshal.AllocHGlobal, tracking their disposal myself (which, for some reason, happened noticeably less often than their creation). By then it was clear that these specific buffers – the ones used for sending and receiving data over the network – were what was leaking. Slowly but surely I kept hunting for the cause, gradually starting to suspect that the 100% CPU load was somehow connected too. In one more desperate attempt, I made Interop calls straight to WSASend/WSAReceive (talking to the network directly on Windows, bypassing .Net entirely), and at that point the memory leaks disappeared. That was very, very strange. I went back to the native .Net socket implementation a few times, and every time the same thing happened: BeginSend/BeginReceive calls didn’t always finish with a matching callback, threads weren’t always returned to the pool, and this was sometimes accompanied by 100% CPU load as well.

The memory stopped leaking, but even that didn’t fix the situation where CPU load would unexpectedly spike while the server kept on running, accepting connections and serving clients, just slowly. Dumping the thread list showed that one of them was stuck dead, running some code and eating 100% of a CPU core – and it wasn’t my code!

Then, in early summer 2005, I read an article about Workstation GC vs. Server GC, and for their sake I switched the server to the .Net 2.0 Preview. That was the breakthrough I’d been waiting for – every asynchronous method call started completing correctly, and the mystery load concentrated itself into a dedicated GC thread that used 5-10% CPU.

I had the time and the mood for it, so I ran a dozen experiments and could say for sure: in .Net 1.1 with Workstation GC, a GC invocation on a Thread Pool thread could cause that thread to hang solid, sometimes even before the called method’s own code ran (hello, buffers never returned to the pool or the heap!), and sometimes after it (hello, 100% load).
Eventually I gave up on BeginSend/BeginReceive entirely and used Windows IOCP exclusively for all network operations. By then the server was fully multithreaded, and connections were handled in parallel, which was quite good and forward-looking, despite the large number of locks that had previously been more or less invisible against the backdrop of OOMs and 100% CPU gluttony.

These days, a server with under 4GB of memory is almost laughable, and bolting on an extra 8-16 gigabytes for a cloud setup takes a couple of clicks and one restart. And yet, when memory started leaking again and CPU load spiked to 100-150% (out of a possible 800% across 8 cores), I felt like a 20-year-old student again, burning gigabytes and gigaflops in the furnace of a ravenous machine. It was strange, abnormal, and frankly dumb. What was especially unpleasant was that, just like before, the game kept running fine (if a bit laggy) – nothing actually broke. Well, not until the memory ran out, of course.

Over the years, Lightweight Threads (aka Fibers) came and went, and as a result we no longer have access to system threads in .Net, only to so-called Managed Threads – and on Mono, there isn’t even access to ProcessThread, just stubs inside. Thread diagnostics got a lot harder, but on the other hand I was now using my own Thread Pool, with every thread accounted for and named, and precise statistics kept for each one – what it’s currently running, how long a given task takes. That let me fairly quickly narrow the fault down to my own code rather than the runtime, and the thread statistics showed the resource hog was tied to business logic execution – some actions were simply running 100 times more often than they should. This time I wasn’t resource-constrained, so I calmly added extra logging to every script and timer invocation, measured the execution time of every event, and after a week of experiments I could say confidently what the problem was. It turned out a certain NPC was trying to attack another NPC, and both of them were stuck in rocks, unable to move, and their attempts to shoot at each other were instantly aborted for lack of line of sight. But on every behavior tick (15ms) they’d still try to path-find and start shooting, and because the shot never actually fired, the weapon’s reload never kicked in either, so the whole thing repeated on the next tick. After a few days of gameplay, hundreds of such NPCs would pile up and end up eating all the server’s resources. The fix was to adjust the behavior and cut down on getting-stuck situations, and to add a small reload time even for failed shots.

And then the server started hanging.

Chapter 3. Frost

Autumn 2005 turned out to be rough – my job situation was up in the air, and my apartment rent suddenly doubled. The only bright spot was the game server, where the player count was already in the hundreds. But trouble started brewing there too – the whole world began to freeze up. At best, pings kept going through, or a few timers kept firing. And sometimes everything froze solid, traffic stopped entirely, and I had to kill the server process and start it again. As before, it was impossible to attach a debugger to the running server because of the heavy resource usage and lag. For some reason, this would simply crash Visual Studio, or hang it.

The problem got more serious by the day – hundreds of players online were furious that they couldn’t play, and threatened to leave. The admins were still just as hard to convince to leave a hung server alone and give me a chance to do something about it. But the trouble was, there wasn’t much I could do. I spent an indeterminate amount of time fighting with SOS.dll. This wonderful library, whose name stands for Son Of Strike, is supposed to work as a WinDbg extension and can show you .Net threads, tasks, exception throw sites, and other useful things. All well and good, except the library is extremely sensitive to the .Net version and GC type. And for some dumb reason, for my specific version, attaching sos.dll worked about one time out of 50. Either the version was wrong, or it attached but wouldn’t cooperate, or Mercury was in retrograde. Literally only once did I manage to see a wall of text and numbers, from which I gathered exactly one thing: we had a deadlock!

Honestly, I wasn’t ready for that. The diagnosis sounded like a verdict – having cross-locking is a clear sign of bad architecture. And then came the usual spiral of self-doubt and gloom: I had little experience, hadn’t read the literature, and had gotten myself tangled up in who-knows-what – instead of building little websites and drinking beer on weekends like a normal person, here he was, elbow-deep in servers! Once I managed to shake off the funk, I started thinking, googling, and hunting for options. First I found the option of using homemade SpinLocks with try/finally around their calls. That doesn’t directly protect against cross-locking, but it does offer a clever way to figure out who’s at fault: threads waiting on a SpinLock don’t block, they visibly burn CPU, so a glance at the thread list and the last task each one ran should, in theory, reveal what everyone’s stuck on. In my case, about 8 threads were stuck at once, so that didn’t help much. Still, I tried an administrative fix: since I’d already switched to my own locking classes, I extended them to keep a separate counter per thread, to know the depth of the “rabbit hole.” In the end, I rewrote from scratch every single place that used two or more locks at once. That took about six months – such problems turned out to be everywhere – but the result was worth it: there were no more deadlocks all the way up to the server’s eventual shutdown.

After that, things also changed on the housing and job front, and with the players chipping in, we bought a new server with a dual Xeon 5130 and 8GB of RAM. The player count climbed to 2000, then briefly touched 2500, before starting to decline. By then, players weren’t satisfied with just a stable server anymore – they wanted features, scripts, talents, and all the rest of the business logic, which I couldn’t keep up with at the speed and quality they wanted. But that’s a whole different story.

On one cold October day in 2020, a planned visit from some live streamers fell through because the server suddenly hung. Authorization still worked, but you couldn’t get into the world, and the Telegram bot stayed silent. A quick check turned up nothing in the logs, no memory problems, and no thread starving for resources. Everything had simply stopped. After muttering something a few times about a glitch in the Matrix and a woman of ill repute, I went hunting for the deadlock. Ever since Microsoft acquired Miguel de Icaza and Xamarin, the Mono documentation has been a sorry sight – it sort of exists, but it’s out of date or leads nowhere. For instance, three quarters of the material on the page about debugging Mono with gdb simply doesn’t apply and doesn’t work. I managed to attach gdb to the frozen server, but commands like call mono_pmip and others gave back gibberish, mostly syntax-error complaints. By some miracle I figured out that gdb wanted me to cast the parameters and results of the mono_* commands to specific types, and with that I finally managed to get a list of threads stuck in a cross-lock. But the numbers in that list didn’t match either the output of ps or the server’s own ManagedThreadId values.
The extended logging I’d built for tracking down the CPU hog turned out to help a lot – it let me see which packets and timers had run last, and I gradually narrowed down my list of suspects. Just my luck, the cross-lock involved not two threads but three, so I couldn’t get a fully detailed picture. That’s when I remembered my old lessons and started going through the code for lock usage. It turned out several refactors over the years had gradually replaced SpinLock with Monitor.Enter/Monitor.Exit, and often with a plain lock. And that’s when I stumbled on an article by Eric Gunnerson, which pointed out you could do this far more simply: use Monitor.TryEnter with a timeout everywhere, and throw an exception if the lock can’t be acquired. It’s an incredibly simple and very effective technique – if a TryEnter call anywhere waits more than 30 seconds and bails out (and the logic has no business taking that long), then that spot absolutely needs investigating, to see who managed to grab the lock object and hang onto it for so long. Kicking myself, I realized I could have cleaned all this up the very same way 15 years earlier, instead of reinventing the wheel with counting “rabbit hole depth.” But maybe, back then, that was for the best.

And then, just as it once did to the emulator, the 4th horseman came for the new project too. Except this time it never even got the chance to become popular. Having three critical problems right at launch took it down fast enough. And the game itself wasn’t exactly mainstream to begin with. But that, too, is a story for another article.

P.P.S. This article uses illustrations by an artist known as Parsakoira, captioned “ChoW#227 :: VOTING :: 4 Horsemen of the Apocalypse,” presumably from the now-defunct site conceptart.com:
https://www.pinterest.com/pin/460141286926583086/
https://www.pinterest.com/pin/490681321879914768/

Originally published on Habr, 2020-11-30.