Webhook Retries and Idempotency

Stash uses Google Cloud Tasks to deliver webhooks, which implements automatic retry logic with exponential backoff. Understanding retry behavior is crucial for building reliable webhook handlers.

How Many Retries

The exact number of retries depends on Cloud Tasks queue configuration, but typically:

• Initial attempt: Immediate
• Retry 1: ~1 minute after initial failure
• Retry 2: ~2 minutes after retry 1
• Retry 3: ~4 minutes after retry 2
• Retry 4: ~8 minutes after retry 3
• Retry 5: ~16 minutes after retry 4
• Maximum retries: Typically 5-10 attempts over ~24 hours

The retry count is included in the stash-retry-count header (see Webhook Headers below).

What Happens on Failure

When a webhook delivery fails after all retry attempts are exhausted:

1. No further automatic retries: The webhook will not be automatically retried again
2. Event is logged: The failure is logged in Stash's internal systems for monitoring
3. Manual reconciliation: You may need to manually reconcile missed events by:
   • Querying the Stash API for transaction status
   • Checking your purchase history endpoints
   • Reviewing transaction logs in the Stash Studio dashboard

Webhook Headers

Each webhook request includes the following HTTP headers:

Content-Type

• Value: application/json
• Description: Indicates the request body is JSON

Stash-Hmac-Signature

• Value: Base64-encoded HMAC-SHA256 signature of the request body
• Description: Used for signature verification (optional but recommended)
• Format: Base64-encoded string
• See: Signature Verification

stash-retry-count

• Value: Integer string (e.g., "0", "1", "2")
• Description: Current retry attempt number (0 = initial attempt, 1 = first retry, etc.)
• Use Case: Track retry attempts, implement retry-specific logic, debugging

You can use the stash-retry-count header to:

• Log retry attempts for debugging
• Implement different handling logic for retries vs initial attempts
• Monitor webhook delivery reliability

Best Practices

1. Idempotent Handlers

Design your webhook handlers to be idempotent (safe to process the same event multiple times). This ensures that if a webhook is retried, you don't accidentally:

• Grant items twice
• Charge a user multiple times
• Create duplicate records

Example:

async function handlePurchaseSucceeded(event) {
  const { orderId, userId, items } = event.purchaseSucceeded;
  
  // Check if this order has already been processed
  const existingOrder = await db.getOrder(orderId);
  if (existingOrder && existingOrder.status === 'completed') {
    // Already processed, return success
    return { status: 'ok', message: 'Already processed' };
  }
  
  // Process the order
  await grantItemsToUser(userId, items);
  await db.saveOrder(orderId, { status: 'completed', ...event });
  
  return { status: 'ok' };
}

2. Use Transaction IDs

Use orderId or transactionId to deduplicate events:

// Use orderId as a unique key
const orderId = event.purchaseSucceeded.orderId;

// Check if already processed
if (await isOrderProcessed(orderId)) {
  return; // Skip duplicate
}

// Process and mark as complete
await processOrder(orderId, event);
await markOrderProcessed(orderId);

3. Reconciliation

Periodically query transaction status to catch missed webhooks:

// Run this periodically (e.g., daily)
async function reconcileMissedWebhooks() {
  const recentOrders = await stashApi.getRecentOrders();
  
  for (const order of recentOrders) {
    if (!await isOrderProcessed(order.id)) {
      // Webhook was missed, process manually
      await handlePurchaseSucceeded({
        purchaseSucceeded: order
      });
    }
  }
}

4. Monitoring

Monitor webhook delivery success rates and set up alerts for failures:

• Track the stash-retry-count header to identify frequently retried webhooks
• Set up alerts for webhook delivery failures
• Monitor your endpoint's response times (should be < 30 seconds)
• Track error rates and response codes

5. Fast Response

Respond quickly (within 30 seconds) to avoid timeouts:

// Good: Process asynchronously and respond immediately
app.post('/webhook', async (req, res) => {
  // Verify signature first
  if (!verifySignature(req)) {
    return res.status(401).send();
  }
  
  // Respond immediately
  res.status(200).json({ received: true });
  
  // Process asynchronously
  processWebhookAsync(req.body).catch(err => {
    console.error('Webhook processing error:', err);
    // Handle error (e.g., queue for retry, send alert)
  });
});

Handling Retries

You can use the stash-retry-count header to implement retry-specific logic:

app.post('/webhook', async (req, res) => {
  const retryCount = parseInt(req.headers['stash-retry-count'] || '0');
  
  if (retryCount > 0) {
    console.log(`Processing retry attempt ${retryCount} for webhook`);
    // You might want to log this differently or handle retries with extra care
  }
  
  // Process webhook...
  res.status(200).json({ received: true });
});

Common Issues

Duplicate Processing

Problem: Webhook is processed multiple times due to retries.

Solution: Implement idempotent handlers using orderId or transactionId as unique keys.

Slow Processing

Problem: Webhook processing takes too long, causing timeouts and retries.

Solution:

• Respond immediately with 200 OK
• Process the webhook asynchronously
• Use background jobs or queues for heavy processing

Missing Events

Problem: Some webhooks are never received even after retries.

Solution:

• Implement reconciliation process
• Monitor webhook delivery logs in Stash Studio
• Set up alerts for delivery failures