How an Integration Failure Becomes a System Outage
Business systems lean on more external services every year: shipping label issuance, payment authorization, public data APIs, ERP journal transfers. The dangerous case isn't when an external API goes down — it's when it gets slow.
Imagine a server with 200 request-handling threads calling a shipping API 20 times per second, and that API starts hanging for 30 seconds per call. Within roughly 10 seconds every thread is stuck waiting, and from then on even unrelated requests like login and search can't be served. If the code holds a database connection while waiting on the external call, the connection pool drains too. That's the classic path from one slow dependency to a full outage.
The Basics: Timeouts and Retries
Derive Timeouts from Business Tolerance
Timeouts should come from how long a user can reasonably wait, not library defaults. If the order confirmation screen must appear within 3 seconds, split it into something like a 1-second connect timeout and a 1.5-second read timeout, and make sure the total — retries included — stays under 3 seconds. Many HTTP clients still default to infinite or multi-second timeouts, so always set them explicitly.
Which Requests Are Safe to Retry
Combine exponential backoff with jitter. Grow the interval (200ms, 400ms, 800ms) and add randomness so many servers don't stampede the provider at the same instant. Cap retries at two or three.
Idempotency to Prevent Duplicate Processing
A read timeout doesn't mean "failed" — it means "outcome unknown." The payment may already have been approved on the other side. Send an idempotency key derived from the order number with every creation request, and keep a request log on your side so the same key can never be registered twice.
Don't blindly resend a request whose response never arrived. Check the result through a status query API first. If the provider offers no such API, track the request in a separate "needs verification" state so someone can reconcile it.
Isolation Patterns
Circuit Breaker
A typical setup: if more than 50% of the last 50 calls fail, open the circuit and fail calls immediately for 30 seconds. Then move to half-open, allow 5 trial calls, and close the circuit if they succeed. Because nothing waits on a dead dependency, threads stay free.
Bulkhead
Give each integration its own thread pool or concurrency limit. If the shipping API gets at most 20 concurrent calls, a stall there leaves payments and ERP untouched.
Moving to Async Queues
Integrations that don't need an immediate answer, such as ERP journal transfers, belong on a queue. Failed messages go to a dead-letter queue and are reprocessed once the provider recovers.
Operations and Observability
In POLYGLOTSOFT SM, our integration checklist regularly covers: explicit timeouts on every external call, idempotency keys on creation requests, circuit breaker and bulkhead settings, dead-letter queue backlog, and per-provider dashboards with alert thresholds.
For systems with many external dependencies, keeping a failure contained at one point isn't optional — it's a baseline requirement. Through our subscription development and SM services, POLYGLOTSOFT assesses the integration layer of your existing systems and helps you put timeouts, idempotency, circuit breakers, and monitoring in place. If integration failures are keeping you up at night, get in touch.