Idempotency in Distributed APIs — Part 1: What It Is and Why It Breaks Everything

You have probably seen this bug before. A user clicks “Place Order” once. The network hiccups. The client retries. Two orders appear. Charges hit twice. Support tickets fly in.

This is not a rare edge case. In any distributed system where networks are unreliable and clients retry, this kind of duplication is the default outcome unless you deliberately design against it. The concept that prevents it is called idempotency, and getting it right is one of the most practical skills you can build as a backend engineer.

What Idempotency Actually Means

An operation is idempotent if applying it multiple times produces the same result as applying it once. The word comes from mathematics — f(f(x)) = f(x). In API terms: sending the same request twice should have the same effect as sending it once.

HTTP already has opinions about this. GET, HEAD, PUT, and DELETE are defined as idempotent by the HTTP spec. POST is not. But that definition only covers the protocol semantics — your actual implementation might not honor it at all.

A DELETE on a resource that no longer exists should return 404 or 204, not 500. A PUT replacing a user’s address should produce the same stored address whether it runs once or five times. These feel obvious when written down, but implementations break them constantly.

Why Distributed Systems Make This Hard

In a single-process application, idempotency is mostly a correctness concern. In a distributed system, it becomes a reliability requirement. Here is why.

When a client sends a request over a network, three outcomes are possible:

• The server received the request, processed it, and the response got back to the client successfully.
• The server received the request, processed it, but the response was lost in transit.
• The server never received the request at all.

From the client’s perspective, outcome 2 and outcome 3 look identical — a timeout. So the client retries. If the server already processed the request (outcome 2), you now have a duplicate operation. Without idempotency, that duplicate causes real damage.

sequenceDiagram
    participant Client
    participant Network
    participant Server
    participant DB

    Client->>Network: POST /orders (attempt 1)
    Network->>Server: request arrives
    Server->>DB: INSERT order
    DB-->>Server: success
    Server->>Network: 200 OK
    Network--xClient: response lost!
    
    Note over Client: Timeout -- retry
    
    Client->>Network: POST /orders (attempt 2)
    Network->>Server: request arrives
    Server->>DB: INSERT order (duplicate!)
    DB-->>Server: success
    Server->>Network: 200 OK
    Network-->>Client: 200 OK
    
    Note over DB: Two identical orders now exist

The network ate the response. The client had no way to know. This is the fundamental challenge — and it gets worse at scale because retries are not optional. They are required for reliability.

Where Things Break in Practice

Let’s be concrete about the failure modes that come up most often.

Payment Processing

A user submits a payment. Your server calls the payment provider, the response times out, and your code retries. If you pass the same amount without an idempotency key, you may charge the customer twice. Stripe, Adyen, and most serious payment APIs require an idempotency key for exactly this reason. Your own payment-adjacent code should too.

Email and Notification Sending

A background job sends a “Welcome” email after a user registers. The job crashes after the email is sent but before it marks the job as complete. On restart, it runs again. The user gets two welcome emails. This is annoying. For transactional emails like password resets or order confirmations, it can be a support nightmare.

Inventory and Stock Changes

A stock decrement endpoint is called twice due to a retry. You now have negative inventory or have sold items you do not have. The downstream effect is either a failed fulfillment or a manual correction that costs real money.

Message Queue Consumers

Most message brokers — Kafka, Azure Service Bus, RabbitMQ — provide at-least-once delivery guarantees. That means a consumer may process the same message more than once. If the consumer operation is not idempotent, you get duplicates in your data or side effects that fire multiple times.

Idempotency vs Safety

Two terms often get confused here. In HTTP, a “safe” method is one with no side effects — GET and HEAD are safe. An “idempotent” method can have side effects, but those effects do not multiply with repeated calls. DELETE is idempotent but not safe — it changes state, but calling it ten times has the same result as calling it once.

POST and PATCH are the ones that need the most care. They are the operations where “run it again” is dangerous.

A Simple Rust Example of the Problem

Here is a naive order creation handler in Rust using Axum. Nothing in this code prevents a duplicate order if the client retries.

use axum::{extract::State, http::StatusCode, Json};
use serde::{Deserialize, Serialize};
use sqlx::PgPool;
use uuid::Uuid;

#[derive(Deserialize)]
pub struct CreateOrderRequest {
    pub user_id: Uuid,
    pub product_id: Uuid,
    pub quantity: i32,
}

#[derive(Serialize)]
pub struct CreateOrderResponse {
    pub order_id: Uuid,
}

pub async fn create_order(
    State(pool): State<PgPool>,
    Json(req): Json<CreateOrderRequest>,
) -> Result<Json<CreateOrderResponse>, StatusCode> {
    // Nothing here prevents a duplicate if this runs twice
    let order_id = Uuid::new_v4();

    sqlx::query!(
        "INSERT INTO orders (id, user_id, product_id, quantity) VALUES ($1, $2, $3, $4)",
        order_id,
        req.user_id,
        req.product_id,
        req.quantity,
    )
    .execute(&pool)
    .await
    .map_err(|_| StatusCode::INTERNAL_SERVER_ERROR)?;

    Ok(Json(CreateOrderResponse { order_id }))
}

If the client sends this request twice — because a timeout triggered a retry — two orders get inserted. The server has no way to know they were the same logical request. We will fix this properly in Part 2 with idempotency keys, but the root problem is clear: the handler is stateless about prior invocations.

What a Fix Looks Like at the Concept Level

The standard solution is to attach a unique client-generated key to each logical operation. The server stores this key alongside the result. On a retry, the server checks if it has already processed this key — if so, it returns the stored result without re-running the operation.

flowchart TD
    A[Client sends request with Idempotency-Key: abc-123] --> B{Server checks key store}
    B -->|Key not found| C[Process request]
    C --> D[Store result against key abc-123]
    D --> E[Return result to client]
    B -->|Key already exists| F[Return stored result]
    F --> E

The logic is simple. The implementation details — where to store keys, how long to keep them, what to do with concurrent requests using the same key — are where the real engineering work lives. We will cover all of that across this series.

What This Series Covers

Over the next seven parts, we will build a complete picture of idempotency in production distributed systems:

• Part 2 — Idempotency keys: design, generation, and storage with Redis and Postgres
• Part 3 — Building idempotent REST endpoints in Rust with Axum middleware
• Part 4 — Retry logic and exponential backoff on the client side
• Part 5 — Idempotency in message queues: Kafka and Azure Service Bus
• Part 6 — Database-level idempotency: upserts, unique constraints, optimistic locking
• Part 7 — Distributed sagas and idempotent compensation
• Part 8 — Production patterns, observability, and monitoring duplicate rates

References

• Stripe API Documentation – Idempotent Requests (https://stripe.com/docs/api/idempotent_requests)
• RFC 9110 – HTTP Semantics: Idempotent Methods (https://datatracker.ietf.org/doc/html/rfc9110)
• AWS Builders Library – Making Retries Safe with Idempotent APIs (https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/)
• Axum Framework Documentation (https://docs.rs/axum/latest/axum/)