.NET Thread Pool and IOCP
In programming we often face the task of doing something “in parallel” with the current flow of execution. There’s a sea of ways to solve this: using extra threads, deferring the task for “later” (manually or via APC), using some kind of task queue, and other forms of asynchronous execution.
What interests us here specifically is the Thread Pool in .NET. Using it is generally considered good practice, and some CLR methods use it implicitly for their own purposes (including System.Net.Sockets and delegates in asynchronous calls). I see only one drawback to it: its universality. People are all different, but factories sew clothes to common patterns and sizes. Every book is unique, but it still gets assigned a category and put on a shelf with others like it..
The problem is that we can’t always live with the fact that all tasks are treated as equals and all get run “as fast as possible.” We occasionally care about how much 25 simultaneously executed data-retrieval tasks will slow down the database, and we’d like to say that a given task can run after all the others. Of course, in 95% of cases this doesn’t matter, but when we’re talking about time-critical applications, such as online game servers, it has to be taken into account. Those same 25 database calls are only efficient if the database server has 25 processors capable of running those queries; otherwise the tasks will either end up queued there, or run in 25 separate threads, which can increase overhead from locking, shared disk access, and so on. In practice, those 25 tasks will almost always finish faster running sequentially than in parallel.
Let’s look at how customizable the Thread Pool actually is. It lets you specify the number of threads (SetMaxThreads/SetMinThreads), and it also calculates the needed number of threads itself based on the number of processors:
The number of threads gives us a degree of control over execution: truly “heavy” tasks are worth running with 2-3 threads, non-time-critical ones in a single thread, and “light” tasks that definitely have no locking are best run with as many threads as there are processors in the system. Unfortunately, all this would only be possible if you could use several pools with different settings. The very beginning of the quote above leaves us no choice – we need to come up with something of our own.
It’s worth looking here at the rather nice IOCP technology: Input Output Completion Ports. As the name suggests, it was developed to ease asynchronous execution of input/output operations. What matters to us is that this technology lets you queue tasks and run them using a specified number of threads. Specifically, we create the threads for the IOCP ourselves and call the GetQueuedCompletionStatus method inside them. That’s where the thread’s execution blocks, and the kernel decides its fate from there. Waiting threads are placed in a LIFO (Last In First Out) queue and resumed as soon as there’s work for them. Using LIFO specifically lets some threads be reused more often than others, which reduces the cost of switching between them.
Implementing your own Thread Pool with IOCP is fairly simple. For the operations you’ll need to pull four functions from kernel32.dll via P/Invoke:
- CreateIoCompletionPort
- CloseHandle
- GetQueuedCompletionStatus
- PostQueuedCompletionStatus
When creating your own pool, you need to initialize an IOCP object by calling CreateIoCompletionPort, then create the required number of threads, each of which calls GetQueuedCompletionStatus in a loop, and then use PostQueuedCompletionStatus to queue a task. The task itself can be passed as a reference to a delegate. That’s not particularly hard to do with the CLR itself (if you specify the delegate as a parameter to PostQueuedCompletionStatus in the P/Invoke declaration), or manually via the Marshal class, or via GCHandle.ToIntPtr/GCHandle.FromIntPtr.
Once you’re done using the IOCP, close it with the CloseHandle method.
This approach gives us the ability to run any number of our own Thread Pools with any number of threads.
Originally published 2008-03-24.