Background tasks in .NET

What is a Background Task?
 

A background task (or background service) is work that runs behind the scenes in an application without blocking the main user flow and often without direct user interaction.

Think of it as a worker or helper that performs tasks independently while the main app continues doing other things.

Problem Statement - 
What do you do when your downstream API is flaky or sometimes down for hours or even days , yet your UI and main API must stay responsive?
Solution -
This is a very common architecture problem in enterprise systems, and .NET gives us excellent tools to solve it cleanly: BackgroundService and exponential backoff retry logic.

In this article, I’ll walk you through:

• A real production-like use case
• The architecture needed to make it reliable
• Why exponential backoff matters
• How to build a robust BackgroundService
• A full working code example

The Use Case

You have two APIs:

• API 1 :  called frequently by the UI (hundreds or thousands of times).
• API 2 : a downstream system you must call, but it is known to be
  unstable, slow, or completely offline for long periods.

If API 1 directly calls API 2:
*  Users experience lag
*  API 1 becomes slow or unusable
*  You overload API 2 with retries
*  Calls fail when API 2 is offline
*  You lose data

What do we do then? Here goes the solution
The Architecture
Instead of calling API 2 synchronously, API 1 simply stores the intended call, and returns immediately.

A BackgroundService will later:

1. Poll for pending jobs
2. Call API 2
3. Retry with exponential backoff if API 2 is still unavailable
4. Mark jobs as completed when successful

This creates a resilient, smooth, non-blocking system.

Why Exponential Backoff?

When a downstream API is completely offline, retrying every 1–5 seconds is disastrous:

• It wastes CPU and bandwidth
• It floods logs
• It overloads API 2 when it comes back online
• It burns resources

Exponential backoff solves this.

Examples retry delays:

Retry 1 → 2 sec
Retry 2 → 4 sec
Retry 3 → 8 sec 
Retry 4 → 16 sec 
Retry 5 → 32 sec 
Retry 6 → 64 sec (Max delay capped at 5 minutes)

This gives the system room to breathe.

Complete Working Example (Using In-Memory Store)

1. The Model

public class PendingJob {
    public Guid Id {
        get;
        set;
    } = Guid.NewGuid();
    public string Payload {
        get;
        set;
    } = string.Empty;
    public int RetryCount {
        get;
        set;
    } = 0;
    public DateTime NextRetryTime {
        get;
        set;
    } = DateTime.UtcNow;
    public bool Completed {
        get;
        set;
    } = false;
}
 

2. The In-Memory Store

public interface IPendingJobStore
{
    Task AddJobAsync(string payload);
    Task<List<PendingJob>> GetExecutableJobsAsync();
    Task MarkJobAsCompletedAsync(Guid jobId);
    Task UpdateJobAsync(PendingJob job);
}
public class InMemoryPendingJobStore : IPendingJobStore
{
    private readonly List<PendingJob> _jobs = new();
    private readonly object _lock = new();
    public Task AddJobAsync(string payload)
    {
        lock (_lock)
        {
            _jobs.Add(new PendingJob
            {
                Payload = payload,
                RetryCount = 0,
                NextRetryTime = DateTime.UtcNow
            });
        }
        return Task.CompletedTask;
    }
    public Task<List<PendingJob>> GetExecutableJobsAsync()
    {
        lock (_lock)
        {
            return Task.FromResult(_jobs.Where(j => !j.Completed && j.NextRetryTime <= DateTime.UtcNow).ToList());
        }
    }
    public Task MarkJobAsCompletedAsync(Guid jobId)
    {
        lock (_lock)
        {
            var job = _jobs.FirstOrDefault(j => j.Id == jobId);
            if (job != null) job.Completed = true;
        }
        return Task.CompletedTask;
    }
    public Task UpdateJobAsync(PendingJob job) => Task.CompletedTask;
}

3. The BackgroundService with Exponential Backoff

using System.Text;
public class Api2RetryService : BackgroundService
{
    private readonly IHttpClientFactory _clientFactory;
    private readonly IPendingJobStore _store;
    private readonly ILogger<Api2RetryService> _logger;
    public Api2RetryService(IHttpClientFactory clientFactory, IPendingJobStore store, ILogger<Api2RetryService> logger)
    {
        _clientFactory = clientFactory;
        _store = store;
        _logger = logger;
    }
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        _logger.LogInformation("Background retry service started.");
        while (!stoppingToken.IsCancellationRequested)
        {
            var jobs = await _store.GetExecutableJobsAsync();
            foreach (var job in jobs)
            {
                var client = _clientFactory.CreateClient("api2");
                try
                {
                    var response = await client.PostAsync("/simulate", new StringContent(job.Payload, Encoding.UTF8, "application/json"), stoppingToken);
                    if (response.IsSuccessStatusCode)
                    {
                        _logger.LogInformation("Job {JobId} processed successfully.", job.Id);
                        await _store.MarkJobAsCompletedAsync(job.Id);
                    }
                    else
                    {
                        await HandleFailure(job);
                    }
                }
                catch (Exception ex)
                {
                    _logger.LogError(ex, "Error calling API 2.");
                    await HandleFailure(job);
                }
            }
            await Task.Delay(TimeSpan.FromSeconds(5), stoppingToken);
        }
    }
    private async Task HandleFailure(PendingJob job)
    {
        job.RetryCount++;
        var delay = CalculateBackoff(job.RetryCount);
        job.NextRetryTime = DateTime.UtcNow.Add(delay);
        await _store.UpdateJobAsync(job);
        _logger.LogWarning("Retrying job {JobId} in {Delay}. RetryCount={RetryCount}", job.Id, delay, job.RetryCount);
    }
    private TimeSpan CalculateBackoff(int retryCount)
    {
        var seconds = Math.Pow(2, retryCount);
        var maxSeconds = TimeSpan.FromMinutes(5).TotalSeconds;
        return TimeSpan.FromSeconds(Math.Min(seconds, maxSeconds));
    }
}

4. The API 1 — Public Endpoint

using System.Runtime.InteropServices;
using System.Text.Json;
[ApiController]
[Route("api1")]
public class Api1Controller : ControllerBase
{
    private readonly IPendingJobStore _store;
    private readonly ILogger<Api1Controller> _logger;
    public Api1Controller(IPendingJobStore store, ILogger<Api1Controller> logger)
    {
        _store = store;
        _logger = logger;
    }
    [HttpPost("process")]
    public async Task<IActionResult> Process([FromBody] object data)
    {
        var payload = JsonSerializer.Serialize(data);
        await _store.AddJobAsync(payload);
        _logger.LogInformation("Stored job for background processing.");
        return Ok("Request received. Will process when API 2 recovers.");
    }
}

5. The API 2 (Simulating Downtime)

using System.Runtime.InteropServices;
[ApiController][Route("api2")] public class Api2Controller: ControllerBase {
    private static bool shouldFail = true;
    [HttpPost("simulate")] public IActionResult Simulate([FromBody] object payload) {
        if (shouldFail) return StatusCode(503, "API 2 is down");
        return Ok("API 2 processed payload");
    } [HttpPost("toggle")] public IActionResult Toggle() {
        shouldFail = !shouldFail;
        return Ok($"API 2 failure mode = {shouldFail}");
    }
}

6. The Program.cs

var builder = WebApplication.CreateBuilder(args);
builder.Services.AddControllers();
builder.Services.AddSingleton<IPendingJobStore, InMemoryPendingJobStore>();
builder.Services.AddHttpClient("api2", c =>
{
    c.BaseAddress = new Uri("http://localhost:5000/api2");
});
builder.Services.AddHostedService<Api2RetryService>();
var app = builder.Build();
app.MapControllers();
app.Run();

Testing the Whole Flow
#1 API 2 starts in failure mode
All attempts will fail and exponential backoff kicks in.
#2 Send a request to API 1
POST /api1/process
{
  "name": "hello"
}
Job is stored.
#3 Watch logs
You’ll see:
Retrying job in 2 seconds...
Retrying job in 4 seconds...
Retrying job in 8 seconds...
...
#4 Bring API 2 back online:
POST /api2/toggle
Next retry will succeed:
Job {id} processed successfully.

Conclusion

This pattern is extremely powerful for:

• Payment gateways
• ERP integrations
• Long-running partner APIs
• Unstable third-party services
• Internal microservices that spike or go offline

References

• Background tasks with hosted services in ASP.NET Core