Implementing cache retries with Enterprise Library to solve intermittent DataCacheException

I have been using Windows Azure Cache for my Facebook application Am I Interesting for quite some time now and have kept getting intermittent exceptions like this one:

Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode<ERRCA0017>:SubStatus<ES0006>:There is a temporary failure. Please retry later. (One or more specified cache servers are unavailable, which could be caused by busy network or servers. For on-premises cache clusters, also verify the following conditions. Ensure that security permission has been granted for this client account, and check that the AppFabric Caching Service is allowed through the firewall on all cache hosts. Also the MaxBufferSize on the server must be greater than or equal to the serialized object size sent from the client.)

Reading up on the Internet about this issue often suggests working with cache objects that are too large, but in my case I have gotten the exception primarily when reading objects that are very small (a couple of hundred bytes). Continuing to read about the Enterprise Library application block on MSDN blogs and on Patterns & Practices, I followed the examples and installed the Transient Fault Handling block using NuGet and updated my CacheHelper class to implement retries. Unfortunately – since I’m already using StructureMap – the install included the DI framework Unity, which Enterprise Library depends on, but so be it…

Cache helper code before implementing retries below. IDataCacheWrapper is a wrapper interface to Windows Azure Cache methods. I use it to inject an in-memory cache when I’m working on my development machine or when deploying to a single instance environment. The code for the concrete class is included further below.

    public static class CacheHelper
    {
        private static readonly IConfigurationHelper ConfigurationHelper;
        private static readonly IDataCacheWrapper Cache;

        static CacheHelper()
        {
            ConfigurationHelper = ObjectFactory.GetInstance<IConfigurationHelper>();
            Cache = ObjectFactory.GetInstance<IDataCacheWrapper>();
        }

        public static T GetFromCache<T>(string cacheKey)
        {
            var cacheItem = Cache.Get(cacheKey);
            if (cacheItem == null)
            {
                return default(T);
            }
            return (T) cacheItem;
        }

        public static void AddToCacheWithNoExpiration(string cacheKey, object value)
        {
            AddObjectToCache(cacheKey,value, TimeSpan.MaxValue);
        }

        public static void AddToCacheWithDefaultExpiration(string cacheKey, object value)
        {
            AddObjectToCache(cacheKey, value, TimeSpan.FromMinutes(ConfigurationHelper.DefaultCacheMinutes));
        }

        public static void AddToCacheWithSpecifiedExpiration(string cacheKey, object value, TimeSpan timeout)
        {
            AddObjectToCache(cacheKey, value, timeout);
        }

        private static void AddObjectToCache(string cacheKey, object value, TimeSpan timeout)
        {
            Cache.AddOrReplace(cacheKey, value, timeout);
        }

        public static void RemoveFromCache(string cacheKey)
        {
            Cache.Remove(cacheKey);
        }
    }

Code changes to implement retries (all other code remains intact):

    public static class CacheHelper
    {
        ...
        private static readonly RetryPolicy RetryPolicy;

        static CacheHelper()
        {
            ...
            RetryPolicy = CreateRetryPolicy();
        }

        private static RetryPolicy CreateRetryPolicy()
        {
            var retryStrategy = new Incremental(3, TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(2)) { FastFirstRetry = true };
            var retryPolicy = new RetryPolicy<CacheTransientErrorDetectionStrategy>(retryStrategy);
            retryPolicy.Retrying += (sender, args) =>
                                        {
                                            var logger = LogManager.GetLogger(typeof(CacheHelper));
                                            logger.WarnFormat("Retrying cache access - Count: {0}, Delay: {1}, Exception {2}",
                                                args.CurrentRetryCount, args.Delay, args.LastException);
                                        };
            return retryPolicy;
        }

        public static T GetFromCache<T>(string cacheKey)
        {
            Object cacheItem = null;
            RetryPolicy.ExecuteAction(() => cacheItem = Cache.Get(cacheKey));
            ...
        }

        private static void AddObjectToCache(string cacheKey, object value, TimeSpan timeout)
        {
            RetryPolicy.ExecuteAction(() => Cache.AddOrReplace(cacheKey, value, timeout));
        }

        public static void RemoveFromCache(string cacheKey)
        {
            RetryPolicy.ExecuteAction(() => Cache.Remove(cacheKey));
        }
    }

And finally the concrete class for Windows Azure Cache that implements IDataCacheWrapper, if any of you need it:

    public class AzureDataCacheWrapper : IDataCacheWrapper
    {
        private readonly DataCache _cache;
        private readonly string _cacheKeyPrefix; // Unique cache objects for each environment

        public AzureDataCacheWrapper(IConfigurationHelper configurationHelper)
        {

            // Cache servers
            var cacheServers = new List<DataCacheServerEndpoint>
                                   {
                                       new DataCacheServerEndpoint(
                                           configurationHelper.CacheHostName, 
                                           configurationHelper.CachePort)
                                   };

            // Cache security
            var secureString = CreateSecureString(configurationHelper.CacheAuthorization);
            var cacheSecurity = new DataCacheSecurity(secureString);

            // Setup configuration
            var cacheConfiguration = new DataCacheFactoryConfiguration("default")
                                         {
                                             SecurityProperties = cacheSecurity,
                                             Servers = cacheServers
                                         };

            var cacheFactory = new DataCacheFactory(cacheConfiguration);
            _cache = cacheFactory.GetDefaultCache();
            _cacheKeyPrefix = configurationHelper.FacebookAppId;
        }

        public object Get(string key)
        {
            return _cache.Get(FullKeyName(key));
        }

        public void AddOrReplace(string key, object value, TimeSpan timeout)
        {
            _cache.Put(FullKeyName(key), value, timeout);
        }

        public void Remove(string key)
        {
            _cache.Remove(FullKeyName(key));
        }

        private string FullKeyName(string key)
        {
            return string.Format("{0}-{1}", _cacheKeyPrefix, key);
        }

        private static SecureString CreateSecureString(string unsecureString)
        {
            var secureString = new SecureString();
            foreach (var character in unsecureString.ToCharArray())
            {
                secureString.AppendChar(character);
            }
            return secureString;
        }
    }

I deployed this code to production earlier today, so if you don’t see an update to this post in the next few days stating some problem, you can assume the solution worked and that my problem is now gone ;-).

Advertisements