Future of Cache in Computer Systems
👷♂️ Software Architecture Series — Part 27
With modern day computers pushing the compute speed every other day, it often leaves one to wonder then do we really need to worry about caches in the system?
Why we need cache in the first place? To reference data quickly or another way of saying would be to speed up memory references. So, what if we make memory cells of our computers as fast as on-chip cache? Will that be the end of the road for Caches?
To get to the bottom of it, let us first understand the physical limitations of a computer system.
Speed of a computer is about how fast it can move information from one place to another, which basically means how fast a computer can move electrons within itself. So, the physical limit of an electron moving through the matter is definitely a determinant in the speed limits of a computer system. And speed of electrons cannot surpass speed of the light. What this means that the when the CPU needs to access data from memory, memory cells closer to the CPU will inherently have lower latency compared to those farther away.
In computer systems, memory modules are typically located at varying distances from the CPU. Memory cells closer to the CPU are physically nearer, while those farther away are physically more distant. And to mitigate these delays, data is often stored in different levels of cache memory to reduce the time it takes for the CPU to access frequently used data. However, even with cache memory, there can still be delays when accessing data from main memory (RAM) or other components of the system, especially if the data needs to travel over relatively long distances on the motherboard or between different components. This delay is called “Speed of Light delays”.
Let us inspect this further.
The speed of light in a vacuum is indeed approximately 299,792,458 meters per second. It is incredibly fast speed; however, it is still limited. In computer systems, we measure speed in terms of clock frequency, that is how many clock cycles occur per second. For example, a system running at 1.8 GHz has a clock frequency of 1.8 billion cycles per second. Light can only manage about an 8-centimeter round trip during a 1.8 GHz clock period. The fastest CPUs these days have a clock speed of about 5 GHz. This round-trip distance drops further to about 3 centimeters for a 5 GHz clock. How do we explain this?
To calculate the round-trip distance that light can travel during the duration of a clock period, we need to consider both the propagation speed of light and the duration of a clock cycle. Since light travels at a finite speed, it can only cover a certain distance within a given time frame. As the clock frequency increases, the duration of each clock cycle decreases. Consequently, light has less time to travel during each clock cycle, resulting in a shorter round-trip distance. This is why the round-trip distance decreases as the clock frequency increases.
Now to make matter worse, electrons don’t flow in vacuum inside computer systems but through silicon. While light travels quickly in a vacuum, electric waves in silicon move much more slowly — typically three to thirty times slower than light in a vacuum. This slower movement of electric waves within silicon further constrains the speed at which signals can propagate within a computer system. For example, a memory reference may need to wait for a local cache lookup to complete before the request may be passed on to the rest of the system. Additionally, relatively low-speed and high-power drivers are required to move electrical signals from one silicon die to another, such as to communicate between a CPU and main memory.
What is implied here that CPU clock frequencies can’t go higher infinitely. Now you can relate why choosing the right location for your data center is important.
Even with faster processors and improved memory architectures, the memory hierarchy — ranging from registers to caches to RAM to storage — will continue to exist. Caches serve as a critical component of this hierarchy by providing faster access to frequently accessed data. While CPUs are getting faster, memory access times are not improving at the same rate. Caches help mitigate the latency gap between the CPU and main memory by storing frequently accessed data closer to the CPU, reducing the need to access slower main memory frequently.
Hardware designers further use techniques such as cache prefetching to minimize the impact of speed-of-light delays, where the hardware reacts to consecutive accesses by prefetching subsequent cache lines, thereby evading speed-of-light delays for these subsequent cache lines. This can be particularly beneficial in systems with high memory access rates, such as those used in high-performance computing or data-intensive applications.
The Future
The physical limits due to speed of light delays in computer systems certainly pose a barrier to increase the speed of computers beyond a certain limit, but scientists are trying to push through these limits via their innovative approaches.
Modern manufacturers are trying to place more and more transistors on to a single chip to further reduce signal processing delays. Some manufacturers are integrating redundant, vital processor circuits in parallel on the same chip. This approach allows multiple phases of data processing to occur simultaneously, increasing the rate of data throughput. By parallelizing processing tasks, chips can handle more computations in parallel, leading to improved performance.
Researchers have taken it further by trying to integrate the entire computer system — including memory, peripheral controls, clocks, and controllers — onto a single centimetre-square piece of silicon. So, we are looking at palm-sized computers way faster than conventional computers we are using today. However, all this being commonly available to public still is a distant future, but nevertheless interesting prospects to keep an eye on!
Further Reading:
https://www.scientificamerican.com/article/computers-are-becoming-fa/