C# Transient Fault Handling With Polly

About

In this code snippet, we’ll look at Polly which is a .NET library used for resilience and transient-fault-handling. It provides you with already implemented software design patterns/mechanisms/concepts such as Retry, Circuit Breaker, Hedging, Timeout, Rate Limiter, Fallback, …You can use this to make your software more reliable.

This is especially useful when dealing with the microservice architecture or making any API call in general including working files, devices or making other types of RPCs(remote procedure calls) in general that could become unavailable or experience an interruption.

I wrote about Azure functions in this post if you are interested. I will utilize Azure Functions for this demo but you can use Polly in any C# project.

Note: For more examples check out the official documentation here.

Install Polly Nuget Package

Install the package from here or just use the NuGet manager within Visual Studio.

Code

using System.Net;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Azure.Functions.Worker.Http;
using Microsoft.Extensions.Logging;
using Polly;
using Polly.Retry;
using Polly.Fallback;

namespace DemoFunctionApp
{
    public class Function1
    {
        private readonly ILogger _logger;

        public Function1(ILoggerFactory loggerFactory)
        {
            _logger = loggerFactory.CreateLogger<Function1>();
        }

        [Function("Function1")]
        public async Task<HttpResponseData> Run([HttpTrigger(AuthorizationLevel.Function, "get")] HttpRequestData req)
        {
            string value = await callAPI();

            var response = req.CreateResponse(HttpStatusCode.OK);
            response.Headers.Add("Content-Type", "text/plain; charset=utf-8");
            response.WriteString(value);

            return response;
        }

        private async Task<string>callAPI()
        {
            //Create a new HttpClient. 
            //This is just for demonstration purposes. For production, register the HttpClient in Program.cs and DI it to avoid web socket exhaustion.
            using HttpClient client = new HttpClient();

            string url = "https://jslaholder.typicode.com/posts/1"; //bad url for simulating a failed request and thereby testing the retry and fallback.
            string fallbackUrl = "https://jsonplaceholder.typicode.com/posts/1";

            #region Define pipline options

            //Create a retry strategy with a custom delay generator and event handler.
            var options = new RetryStrategyOptions<HttpResponseMessage>
            {
                ShouldHandle = new PredicateBuilder<HttpResponseMessage>().Handle<HttpRequestException>(),
                BackoffType = DelayBackoffType.Exponential,
                MaxRetryAttempts = 5,
                DelayGenerator = static args =>
                {
                    var delay = TimeSpan.FromSeconds(Math.Pow(2, args.AttemptNumber));

                    return new ValueTask<TimeSpan?>(delay);
                },
                OnRetry = static args =>
                {
                    Console.WriteLine($"OnRetry, Attempt: {args.AttemptNumber} at {DateTime.Now}");

                    // Event handlers can be asynchronous; here, we return an empty ValueTask.
                    return default;
                }
            };

            //Create a fallback strategy with a custom fallback action.
            var optionsOnFallback = new FallbackStrategyOptions<HttpResponseMessage>
            {
                ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
                .Handle<HttpRequestException>(), //.HandleResult(r => !r.IsSuccessStatusCode)
                FallbackAction = async args =>
                {
                    var response = await client.GetAsync(fallbackUrl, CancellationToken.None);

                    Console.WriteLine($"Fallback response");

                    return Outcome.FromResult(response);
                }
            };

            #endregion


            #region Create the pipeline

            //Create an instance of builder.
            var pipelineBuilder = new ResiliencePipelineBuilder<HttpResponseMessage>();

            //Add the retry and fallback policies to the pipeline.
            pipelineBuilder.AddFallback(optionsOnFallback);
            pipelineBuilder.AddRetry(options);

            //Build the pipeline.
            var pipeline = pipelineBuilder.Build();

            #endregion


            #region Run the pipeline 

            //Create a cancellation token that can later be used to cancel the request if needed.
            CancellationToken cancellationToken = new CancellationToken();

            //Execute the pipeline.
            HttpResponseMessage result = await pipeline.ExecuteAsync<HttpResponseMessage>(
                async cancellationToken => 
                { 
                    return await client.GetAsync(url, cancellationToken);
                }, 
                cancellationToken
            );

            #endregion

            //Read response as a string and return it.
            return await result.Content.ReadAsStringAsync();
        }
    }
}

Result

As you can see in the logs the calls to the primary(intentionally bad URL for the demo) have failed. The call was repeated 5 times as specified in the retry strategy options and after each failure the time between calls was increased.

After falling 5 times the fallback strategy was executed and the API call was made to the secondary URL which returned a successful result.