I/O Completion Threads in .Net

I’d wanted to write an article about the quirks of .Net Sockets for large-scale servers for a while, but kept putting it off “for later.” I want to warn upfront that everything described here concerns TCP.
The first and most important quirk of these sockets is that the asynchronous calls (BeginSend/BeginReceive), despite a widespread misconception, don’t use IOCP at all. Instead, an event gets created (essentially a WaitHandle) that waits for data to show up on the socket and then invokes the callback through the ThreadPool. Interestingly, if there’s already data on the socket, the callback fires immediately, in the same thread as BeginReceive — but the documentation doesn’t say a word about this. In local-network tests that’s usually exactly what happens, so technically you might never even see the ThreadPool spawning threads — and by default up to a thousand (!) of them are allowed per processor:

The thread pool has a default size of 250 worker threads per available processor, and 1000 I/O completion threads

The second quirk has to do with some internal peculiarity of how sockets interact with the ThreadPool. It’s fairly non-obvious and you won’t spot it in preliminary tests. The callback code for BeginReceive usually looks something like this:

private void OnReceive(IAsyncResult asyncResult)
{
    SocketError errorCode;
    int size = m_socket.EndReceive(asyncResult, out errorCode);

    if (errorCode != SocketError.Success)
    {
        OnError(new SocketException((int) errorCode));
        return;
    }

    OnData(m_buffer, size);
    m_socket.BeginReceive(m_buffer, 0, m_buffer.Length, SocketFlags.None, OnReceive, null);
}

It all looks logical and correct. But in practice, some fascinating edge cases can happen. As long as there’s data in the buffer, the callback keeps firing on the same thread as BeginReceive. That could be the main program thread or even one of the I/O Completion Threads — nothing criminal about that. The trouble starts when there’s no data in the buffer and we’re no longer on the program thread but on one of the temporary ones. The wait event binds itself to the current thread and won’t release it back to the pool. A new thread gets pulled from the pool for the next call, and only after EndReceive does the previous thread go back to the pool. In practice, with a few hundred connections, you get a constant churn of dozens (sometimes hundreds) of threads being pulled out and returned to the pool. With gcServer=false this is also prone to triggering garbage collection right inside the body of a temporary thread, which in .Net 1.1 and 2.0 (before SP1) could cause that thread to hang for a long time.
I’ve only found one way to fight this affliction: never call BeginReceive from a temporary thread. For each connection we keep a Receiving flag that’s set while a receive is in progress; a separate thread regularly scans the collection of sockets and calls BeginReceive for the ones without the flag set. The downside of this approach is obvious — the more connections there are, the longer the gap between an EndReceive and the next BeginReceive can get. This is partly mitigated by checking asyncResult.CompletedSynchronously. If it’s true, the call was synchronous and it’s safe to call BeginReceive straight from the callback.

The third quirk isn’t relevant all the time or for everyone. The gist of it is that whenever you hand a byte buffer to BeginSend/BeginReceive, you’re condemning it to a hard fate: to call the underlying system functions (WSASend/WSARecv on Windows), the buffers get fixed in memory — the term for this is pinned. Such objects drop out of the GC’s view for a while, can’t be moved, and lead us straight down the road to memory fragmentation. This matters both for small allocations and for large ones. The best way to fight this is pooling. Maoni Stephens says much the same thing:

if you are using the raw socket I/O you do have control over the buffers so you can have a buffer pool and reuse buffers

Unfortunately a pool can’t be one-size-fits-all — at one moment we might need 10,000 buffers, and at another only 500, and there’s no point keeping all 10,000 allocated the previous time around. The pool size, the minimum and maximum buffer sizes, and so on need to be tuned to the specific workload and load, or you need to implement automatic balancing.
In our test projects, using .Net Sockets shows a heavier load on the GC, which even with a well-balanced pool is noticeably higher than with native IOCP. With gcServer = true this load is fairly small (4-5% at 1000 connections on our test server), and on top of that it falls on dedicated threads and doesn’t affect the overall workflow.

Among the upsides of .Net Sockets, I’d point out the fairly convenient API for working with them, and lower latency than calling C++ methods from an external DLL.
Starting with version 2.2, RunServer clients will be able to try out and compare these two technologies without any major changes on the high-level side.