Idea of multi cores
Instead of having one Execution unit (ALU), one Fetch/Decode unit, some registers and a giant cache on the processor. Maybe we can reduce the cache and have two of those earlier components to make two processors. Even if the two units are 25% slower, if we use both of the cores, we’ll be running at (speedup!)
SIMD
If we have a two-nested loop program computing something, and the outer loop is completely parallelizable, we can push the iterations to different processors and scale out efficiently. In this case, we can notice that all the processors are running the same instructions (the loop body is same) just with different data (registers)
So: what if we had one Fetch/Decode unit and have multiple Execution units so that one instruction could compute multiple things. Single Instruction, Multiple Data (SIMD)! The registers then become vector registers (256 bits, can hold 8 * 32-bit bits) and the processor can work on these registers
We can code using AVX intrinsics to use these vector registers in C. With 16 SIMD cores, we can process 128 elements in parallel as each core is processing 8 elements at once.
Conditional Execution
How does a SIMD processor run code in branches when some elements of the vector need to run one branch and the other need to run the other branch?
We can run both of the branches and mask out the effects so that the correct branch runs on correct data. This is same as how branches in CUDA programming work!
Superscalar vs Multicore vs SIMD
- Superscalar: Multiple fetch/decode units along with registers in a core. Not sure why more registers are required?? Or how they’ll be used.
- Multicore: Replicate same core multiple times
- SIMD: Vector registers and instructions so you can compute on multiple data at the same time.
Multithreading to reduce stalls
Here we have:
- 1 Fetch/Decode unit
- multiple Execution units (registers)
- Single ALU
Here we can process multiple instruction streams such that when instruction stream hits a stall like waiting for data from RAM, we can immediately switch to another execution unit and process that. If we had 4 Execution Units, we’ll have 4 threads that can consume the core and this mechanism keeps the utilisation high even with stalls.
Note that this maximises the throughout overall but one thread might take longer to complete since it might have to wait for some time after it’s runnable for the CPU to get free.