System Cache

In the world of web development, performance optimization is a critical aspect that directly influences user experience and overall system efficiency. Caching is a technique that helps to mitigate the load on backend services, reducing response times and enhancing the user experience. One particular approach, the System Cached List, offers a smart and efficient way to cache data and deliver it seamlessly to users. In this article, I will discuss the System Cached List and explore its advantages and disadvantages.

The Challenge: Sub-Second Response Times for a Weather API

You're running a Web API that provides weather forecasts to a large number of users. Each forecast request takes approximately 10 seconds to complete. Your consumers expect near-instantaneous responses and your API is handling thousands of hits every minute.

Caching Strategy:

Data Retrieval:

• When a weather forecast request comes in, the API checks the cache for existing forecast data.
• If cached data is present the API directly serves the cached data to the user.

Cache Miss:

• If the cache is empty, the API waits for the weather forecast data to be retrieved.
• If the cache is past the refresh time, the API serves the stale data to ensure a quick response and backend refresh is initiated.

Background Cache Refresh:

• The API kicks off an asynchronous cache refresh process, updating the forecast data in the cache for future requests.
• This refresh process does not block the main thread, ensuring the API remains responsive.
• A static boolean is used to ensure only one refresh process is running at a time.

Understanding the System Cached List

The System Cached List is a caching mechanism that maintains a cached dataset, periodically updating it while ensuring minimal impact on the application's performance. This technique employs the power of the MemoryCache class and a custom CachedData class to manage cached data efficiently.

I used a minimal Api sample project to start this solution. I then added a generic implementation of caching a list of T using system memory as the caching mechanism. You can view the source for this article in my sandbox repository on GitHub. WebApiCache

Advantages of the System Cached List

Improved Performance:

One of the most significant advantages of the System Cached List is its ability to significantly improve performance by reducing the load on the backend. Instead of hitting the data source every time a request is made, cached data is provided, resulting in faster response times and a smoother user experience.

Reduced Latency:

With cached data readily available, there's a notable reduction in latency. Users experience quicker load times, leading to higher satisfaction and engagement levels.

Efficient Resource Usage:

By utilizing memory caching, the System Cached List optimizes resource usage. Cached data remains in memory, reducing the need for repeated database queries and subsequent resource consumption.

Customizability:

The approach allows customization of cache expiration times. Different datasets might have varying levels of volatility, and the ability to set cache durations accordingly ensures freshness when needed while avoiding unnecessary updates.

Concurrency Handling:

The use of synchronization locks in the implementation prevents potential concurrency issues when updating the cache. This ensures that only one thread updates the cache at a time, maintaining data integrity.

Graceful Update Process:

The System Cached List employs a smart update process. Instead of blocking the application during data update, it runs the update process asynchronously, thus avoiding any negative impact on user interactions.

Disadvantages of the System Cached List

Stale Data Risk:

Depending on the cache expiration settings, there's a risk of serving stale data to users if updates are infrequent or delayed. Careful consideration of cache expiration times is essential to strike the right balance between freshness and efficiency.

Increased Complexity:

Implementing a System Cached List requires careful consideration of thread safety, cache management, and update scheduling. This complexity can sometimes lead to potential bugs or challenges in maintenance.

Memory Usage:

While memory caching is efficient, it's essential to monitor memory consumption. Storing large datasets in memory can lead to increased memory usage, potentially affecting the overall system's performance.

Cold Start Overhead:

When the application starts or the cache expires, there might be a slight overhead as the cache is repopulated. This initial delay could impact the user experience momentarily.

Implementing the System Cached List

The provided code snippet showcases the implementation of the System Cached List. The primary components include:

CachedData Class:

This class encapsulates the cached data along with metadata such as the last update time and the expected next update time.

class CachedData
  {
    public DateTime LastUpdated { get; set; }
    public DateTime NextUpdate { get; set; }
    public object Data { get; set; }
  }

SystemValuesCache Class:

This static class acts as the core of the caching mechanism. It initializes a MemoryCache instance and provides the GetCachedData method to fetch and manage cached data. It also uses a synchronization lock to ensure thread safety.

The GetCachedData method's fetchDataFunction argument plays a crucial role in the functionality of the System Cached List. It is a delegate (a type that references a method) that defines a function responsible for retrieving the data that will be cached. This argument essentially tells the caching mechanism how to obtain the fresh data that needs to be cached.

In the context of the System Cached List, the fetchDataFunction argument is a callback function provided by the caller of the GetCachedData method. This function is expected to perform the actual data retrieval from the data source, which could be a database, a web API, a file, or any other data repository. The function is asynchronous (Task<List<T>>), indicating that it can be executed in a non-blocking manner, allowing the application to continue functioning while the data retrieval is in progress.

public static CachedData<T> GetCachedData<T>(
  string cacheKey,
  Func<Task<List<T>>> fetchDataFunction,
  double cacheTimeInSeconds)
{
  List<T> cachedValues = _cache.Get(cacheKey) as List<T> ?? new List<T>();
  if (cachedValues.Count == 0 || DateTime.Now - _lastUpdateTime > TimeSpan.FromSeconds(cacheTimeInSeconds))
  {
    lock (LockObject)
    {
      Task.Run(async () =>
      {
        var data = await fetchDataFunction();
        cachedValues.Clear();
        cachedValues.AddRange(data);
        var cachePolicy = new CacheItemPolicy
        {
          AbsoluteExpiration = DateTimeOffset.Now.AddSeconds(cacheTimeInSeconds)
        };
        _cache.Set(cacheKey, cachedValues, cachePolicy);
        _lastUpdateTime = DateTime.Now;
      }).Wait();
    }
  }
  return new CachedData<T>()
  {
    Data = cachedValues,
    LastUpdateTime = _lastUpdateTime,
    NextUpdateTime = _lastUpdateTime.AddSeconds(cacheTimeInSeconds)
  };
}

Conclusion

The System Cached List is one approach to enhance the performance of web applications by managing and serving cached data. It combines the efficiency of memory caching with smart update strategies, ensuring users receive quick responses while reducing the load on backend resources. While it offers numerous advantages, it's crucial to carefully consider cache expiration times and handle potential complexities to reap the full benefits of this caching technique.