last article i covered OS basics which got good support and had folks who worked @amazon @lyft dm me about it, continuing off this in this article for inference engineering im covering how we can make a process multi-threaded and introduce concurrency https://t.co/FWIRE5SeKI

Inference Engineering: Concurrency & Threads | Part 2

In our last article we covered what are processes (a running program). Just like how a process is an abstraction of a program, threads are an abstraction of a process. The only difference being that processes share a different address space but threads share the same address space.

Harshit Nayan@HarshitNay80531·Sep 13So I decided to study inference engineering and started with the fundamentals first. Studied about how the operating system abstracts the programs through processes. ArticleInference Engineering: OS Basics | Part 1We run many processes at the same time on our computer (like browser, spotify, codex). The primary question here is how does an OS make a limited resource (like CPU) seem unlimited or rather provide…96557033K

You can think of a classic process as a single-threaded process. But the thing is, a process can be multithreaded. Let us visualize it through a diagram so it is more clear to us. On the left side you can see a classic process with a single thread and on the on the right side you can see a multi-threaded process which has 2 threads in it T1 and T2.

Single-Threaded And Multi-Threaded Address Spaces

Single-Threaded And Multi-Threaded Address Spaces

As you can see through the image the number of stacks in an address space depends on how many threads there are in a process. On the right side you could see that the stack and heap could grow independently and we had a problem only when we ran out of space in the address space. But in a multi-threaded process you could see that size availability of thread 2 (T2) is dependent on thread 1 (T1). So now you know how do we get that classic error of stack overflow, which happens when this free space in the available stack is filled completely.

Okay but why should we use multi-threads? Why can’t we just make a simple process with a single thread? Because it gives us:

• Parallelization

• Non blocking programs

Parallelization and concurrency can be confusing as they are very similar. I’ll try to explain it through the two diagrams.

• Concurrency: For example, imagine you have one CPU: Only one thread is actually running at any given instant, but the CPU switches between them.

concurrency

concurrency

2. Parallelization: With two CPUs: Thread A and Thread B can literally execute at the same time.

parallelization

parallelization

Okay wait, hold up. So from what we know, multi-threads occupy the same address space, right? Then in the above example thread A and thread B will have access to the same address space. And since they can execute at the same time, what happens when they try to update a shared variable at the same time? This is where things get slightly confusing.

Let’s take a simple example. Suppose we have a shared variable counter = 0, and both threads want to increment it 5 times.

Thread 1: counter = counter + 1 (5 times) Thread 2: counter = counter + 1 (5 times)

We would expect the final value to be 10. But that’s not necessarily what happens. The reason is that counter = counter + 1 isn’t actually one operation. It is roughly:

Read counter Add 1 Write counter

For example, let’s say counter = 5. Thread 1 reads 5, then before it can write 6, the OS switches to Thread 2.

T1: read 5 T2: read 5 T2: write 6 T1: write 6

Both threads incremented the counter, but the final value is only 6 instead of 7.

This is called a race condition. The result depends on the timing of the threads.

This is called a race condition. The result depends on the timing of the threads.

Okay so what we get from this is that computers are not really deterministic. Well this seems kind of odd, right? The problem isn’t that the computer is random. The problem is timing.

When multiple threads are running, we don’t know exactly when the OS will switch from one thread to another. One time it might switch here, another time it might switch there. So the instructions themselves are deterministic, but the order in which concurrent threads execute those instructions can vary.

How do we solve this problem? In order to solve this problem we must first understand what exactly this problem is caused by: shared memory state by threads (refer to our first image).

The part of the code that accesses this shared memory state is called a critical section. So we would want only one thread to execute this section at a time. This is called mutual exclusion.

But how do we actually make sure that only one thread enters the critical section at a time? The simplest idea would be to say: “Just check if another thread is already inside. If not, enter.” But there’s a problem. The check itself can be interrupted. Imagine two threads doing this:

T1: check → nobody is inside T2: check → nobody is inside T1: enter critical section T2: enter critical section

Now both threads are inside the critical section, and we’re back to the same problem. So we need an operation that cannot be interrupted halfway through. We need something **atomic **which is a fancy way of saying “all or nothing” operation. This is the basic idea behind the synchronization mechanisms. One common solution is a lock().

So instead of:

T1: read → modify → write
T2: read → modify → write

we get:

T1: lock → read → modify → write → unlock T2: wait → lock → read → modify → write → unlock

**How do we implement the lock itself? **Because if acquiring the lock isn’t atomic, we could end up with the exact same race condition we were trying to solve. At the hardware level, CPUs provide special atomic operations that let us check and change a value without another thread interfering in between.

We don’t really need to worry about the exact implementation for now. The important idea is: **hardware gives the OS the building blocks needed to create safe locks.

**So by now we’ve developed a pretty good mental model for threads → enable concurrency and parallelism → shared state creates synchronization problems → race conditions → critical section → mutual exclusion → atomicity → synchronization mechanisms.