Socket Exhaustion

Socket Exhaustion

Context: A few lines of code that belong outside of a for-loop can crash your entire service, and mislead your most senior talent into thinking they need major server scaling upgrades despite dealing with fewer than millions of records.

In its purest form, the problem looks like this:

# Bad: Creates new socket for each API call
for item in batch_items:
    client = ApiClient()  # New socket every iteration
    client.send(item)

In theory, a server provides 65k+ sockets available per process on a linux system, and perhaps over 1 million for the entire system, but your specific system probably has a default configuration (run ulimit -n on linux) of 1024 per process. You also cannot assign all 65k to ports. Despite the correct CompSci answer being "65k", the reality of your system's configuration probably only provides 16k, 24, or 60k available to you. And if you ignore that limitation, your system will hit other constraints far sooner, such as kernel memory limits, or network buffer pool limits, in-app memory limits, or even some artificial library/framework-imposed limits you might not know about.

In its purest form, the problem looks like this:

# ❌ Bad: Creates new socket for each API call
for item in batch_items:
    client = ApiClient()  # New socket every iteration!
    client.send(item)

Database Connections

We will discuss HTTP APIs primarily, but variations of this issue can exist for files open, unix sockets, or any TCP/IP connection like to a database or redis.

The second most common defect in many projects not using a modern ORM (or perhaps mis-using an ORM) is they can just as easily obscure logic that spawn TCP/IP connections to the database host, causing the app server to launch a sort of denial-of-service attack on your database. Ideally your database, redis, kafka, rabbitmq, or other infrastructure has a way to manage not only the requests it makes but the number of connections too.

How the project misbehaves

The pattern: Everything works perfectly, then suddenly the entire system freezes. Not gradually slower, but completely unresponsive. Sometimes there is a warning of instability when request failures begin to "ramp up".

The distinctive characteristic of socket exhaustion:

• Complete system freeze (not just slowness)
• Affects everything using sockets simultaneously (web requests, database, Redis, monitoring)
• Recovery happens for no apparent reason after a minutes of waiting, or a system restart (socket timeout cleanup)
• Cannot reproduce with small data volumes
• Happens at unpredictable times based on workload volume

Common symptoms:

• "The website just stopped responding". Some slow responses, some HTTP 504 responses at the maximum timeout of your forward proxy, but possibly even requests that just never get a response (not even a 500).
• Random requests fail based on which backend server receives them (if load balanced)
• Database connection errors appear suddenly despite typical workloads.
• Monitoring gaps in the logs. Your APM itself loses connection and shows data gaps
• Cannot SSH into the server during the incident
• Dev/test environments work fine (lower volume doesn't trigger limits)
• Background jobs appear "stalled" mid-execution, or do not complete the entire batch.
• Often misdiagnosed as: high CPU, memory issues, "network hiccup", rate limiting, backup interference

Misleading logs

When experiencing socket exhaustion your server logs might indicate something misleading. For example, if your database ORM cannot find the database due to socket exhaustion it might report issues with queries taking too long, making it look like the DB is running slow. In reality the server has no sockets left and the error stack-trace in your ORM's code does not report this error correctly, leaving misleading messages.

Patterns that can easily create this problem

The trap: The bug hides behind abstractions. Even when aware of this issue, code reviews can easily miss it because the loop and client creation can be in distant unrelated parts of the code.

This issue appears most commonly when:

• Loop + client initialization together: The most obvious case. Creating API clients, database connections, or file handles inside loops
• Abstraction hides the loop: A function called in a loop creates its own client internally. Example: for user in users: send_email(user) where send_email() creates a new SMTP client each time
• Framework/ORM magic: Your ORM opens a new database connection per query when auto-commit is enabled, or your framework instantiates new HTTP clients per request
• Multi-feature implementations: Multiple developers working on different features all hit the same API, each implementing their own client. No individual feature breaks the socket budget, but collectively they exhaust sockets
• Misunderstanding object lifecycle: Not knowing that requests.Session(), psycopg2.connect(), or open() create system resources that must be managed

On a higher-order, they frequently are triggered by features such as:

• Webhook handlers: Each incoming webhook can spawns a new client to call another service
• Scheduled bulk jobs: data sync tasks, email campaigns, inventory updates ... anything processing thousands of items
• Event/message consumers: Processing Kafka/RabbitMQ messages without reusing or managing client sockets

The code works fine in development because:

• Test data volumes are 10-100 items, production is 10,000+
• Dev environments mock external services to avoid putting test data into production systems (leading to no real sockets in use)
• Local database is same-host or routed differently by the OS (sockets recycle instantly, different limits)

What the code problem looks like

You can see how this problem exists in multiple languages and API request libraries using this interactive tool:

Node.js + AxiosNode.js + GotPython + RequestsPython + aiohttpPython + urllib3Go + http.ClientJava + OkHttpJava + Apache HttpClientC# + HttpClientPHP + cURLRuby + Net::HTTP

// Broadcasting urgent alerts to 50k usersconst users = await db.query('SELECT * FROM users WHERE notifications = true'); // ~50,000 users for (const user of users) { const client = axios.create({ timeout: 5000 }); // NEW SOCKET!  try { await client.post('https://api.pushservice.com/send', { // API CALL! userId: user.id, message: 'System maintenance in 30 minutes', priority: 'urgent' }); } catch (error) { console.error('Push failed:', error); }}

System Sockets

Kernel Memory (Socket Structures)

Network Buffer Pool

System Ports

Data Transmitted

Data Received

The fix is fairly painless, just moving a few lines above the for-loop like this:

# ✅ Good: Reuse single socket
client = ApiClient()  # One socket for all calls
for item in batch_items:
    client.send(item)

You can also run through simulations of the corrected code:

Node.js + Axios (Optimized)Node.js + Got (Optimized)Python + Requests (Optimized)Python + aiohttp (Optimized)Python + urllib3 (Optimized)Go + http.Client (Optimized)Java + OkHttp (Optimized)Java + Apache HttpClient (Optimized)C# + HttpClient (Optimized)PHP + cURL Multi (Optimized)Ruby + Net::HTTP (Optimized)

// Broadcasting urgent alerts to 50k users - OPTIMIZEDconst users = await db.query('SELECT * FROM users WHERE notifications = true'); // ~50,000 users // CREATE CLIENT ONCE - REUSE CONNECTION POOLconst client = axios.create({ timeout: 5000, maxRedirects: 3, // Axios automatically uses HTTP keep-alive and connection pooling}); for (const user of users) { try { await client.post('https://api.pushservice.com/send', { // REUSE CONNECTION! userId: user.id, message: 'System maintenance in 30 minutes', priority: 'urgent' }); } catch (error) { console.error('Push failed:', error); }}

System Sockets

Kernel Memory (Socket Structures)

Network Buffer Pool

System Ports

Data Transmitted

Data Received

Parallel Requests

Some developers want to create multiple API clients to manage multiple simultaneous requests to reduce the time it takes to process all API calls.

This is better done through "connection pooling". Many modern HTTP Request libraries offer a way to pool and re-use sockets avoiding the need to constantly open and close connections, or grow connections to numbers that crash the server.

Often the socket exhaustion problem is obvious and easy to fix. However, it can be obscured deep through multiple layers of indirection and business objects.

Imagine an OrderProcessor and a DiscountCalculator. Each order has 1-5 items on it, and the DiscountCalculator is called by OrderProcessor to ensure the price is always up-to-date.

class OrderProcessor
    def refreshPrices():
        for order in orders:
            DiscountCalculator.apply(order)

Recently DiscountCalculator has been upgraded to integrate with a new service that needs API requests to calculate the price of an item.

class DiscountCalculator
    def apply():
        priceIntegration = new APIClient() # This is some NEW API integration
        for items in order:
            priceSystem = items.getSystemPrice();
            priceIntegration = priceIntegration.getIntegrationPrice()
            item.updatePrice(priceSystem, priceIntegration)

Even though it seems that priceIntegration ran the socket-creating code outside the for-loop, it is still called inside another for-loop from another module. OrderProcessor always ran through all open orders to "refresh" values to the most recent state, but now it spams hundreds of API connection sockets.

So while in many cases it is easy to spot, it can also be obscured behind multiple layers of indirection, or developers could incorrectly assume a deeper abstraction manages the connection pool to avoid this problem.

These problems are not about the use of underpinning technology like kafka vs rabbitmq, REST vs protobuf or whatever you picked. It is purely about how you write a few lines of code that manage sockets indirectly. You can have this problem in any arbitrary technology you pick. You do not need to be junior to have this issue either.

How to fix it

Move the API Client object initialization call outside of the for-loop as discussed above. You may wish to use connection pooling in your API Clients if the library provides it.

Do that first. Move the client out of the loop.

If you do not have a connection pool baked into your library, it can easily be added on-top with a small wrapper. This could look like an API Wrapper class that calls the original API client, and inside the wrapper is a singleton connection pool.

Details

Wrapping Python Requests

As an example, imagine this code in python requests:

import requests

def spam_api_clients(urls):
    for url in urls:
        client = requests.Session()
        response = client.get(url)

Sometimes API clients can try to make the request when there is no assigned client, leading to the same issue with even less obvious code and no errors reported:

import requests

def spam_api_clients(urls):
    for url in urls:
        response = requests.get(url) # no `Session()` is used, so it creates one per `get(url)` under-the-hood.

Most modern HTTP libraries have built-in connection pooling:

• Python requests: Use requests.Session() and configure HTTPAdapter
• Python httpx: Connection pooling enabled by default with httpx.Client()
• Node.js axios: Configure httpsAgent with keepAlive: true
• Go net/http: Use http.Transport with MaxIdleConns
• Check your library's documentation.

For emergency fixes, you can create a singleton wrapper around your client to ensure a pool is always used. However this should be temporary until you properly implement connection pooling. A bespoke API Client can be useful in specific contexts, but generally it should be thought out in more depth than possible in a bug-fix.

# Simple connection pool wrapper. Illustrative only.
import requests

class APIClientPool:
    _session = None
    
    @classmethod
    def get_session(cls):
        if cls._session is None:
            cls._session = requests.Session()
        return cls._session

# Usage in your code
for url in urls:
    session = APIClientPool.get_session() # APIClientPool Reuses one session under-the-hood.
    response = session.get(url)

Production Considerations

This example above is for illustration only and does not handle thread safety or connection pooling configuration. For production use:

• Use your library's built-in pooling - Most HTTP libraries (httpx, urllib3, aiohttp) have thread-safe connection pooling by default
• For multi-threaded applications - Use threading.Lock() around session initialization, or threading.local() for per-thread sessions
• Configure pool sizes - Set appropriate pool_connections and pool_maxsize for your workload

The emergency fix that matters most: move client creation outside your loops. Optimize thread safety and pool configuration after service is restored.

As an emergency fix, there are ways to patch the base API Client globally in most cases to pool sessions, avoiding needing to directly change the lower-level code elsewhere (but this can be risky):

# Replace the original requests module with a pooling-enabled 
# api-compatible wrapper around it:
import sys
sys.modules['requests'] = _RequestsWrapper()

It is always better to ship a simple basic "move the client above the loop" code first as it takes minutes to change and easy to roll back if something unexpected happens. And adding a pooling wrapper might take a few hours, and if you are unlucky it can take the whole day or longer. Your near-term goal is to return service, your next goal is to make it work better. Get the business back online first.

Beware of cross-feature for-loops in business logic.

As shown earlier, due to indirection and abstraction they are not obvious. Many mature projects will not be able to refactor cross-feature usage of APIs, but if you create a wrapper that preserves the same client API and simply forwards API calls to the original API Client, and use a singleton connection pool, you could use a drop-in replacement that avoids turning this into a major refactoring task -- simply updating an import statement. Also note that for a trauma fix you might only need to update the most critical problem points, not the whole system.

If you see API query tasks complete significantly faster than before, do verify that requests run correctly but do not be too surprised. Without the constant overhead of adding new objects and sockets your software can execute much faster. As mentioned earlier with the Kafka example, hours and high failure became seconds with 100% success.

Truly massive requests

Even if you have a truly massive request volume to manage, do not underestimate the massive amount of data you should be able to send on a modern computer, and do not under-estimate where the bar for "massive" is.

A modern web server should be able to handle 10k to 50k requests per second for basic CRUD operations. Even if we make your operations 10x or 100x more complex, most people reaching for horizontal workload scaling do not fit in the space of 100 requests per second.

I frequently see teams reach for scaling solutions to manage hundreds of thousands of API requests per day, but this is a far cry from something that needs to be distributed given modern hardware and performant coding practices. Hitting a wall at this scale is a problem for software performance, not a hardware constraint to solve with multiple systems. You may wish to have additional servers for redundancy or quality of service, but it is a very slow and expensive way to overcome simple-to-correct code. Scaling API request workload horizontally will perpetuate many of your existing performance challenges and add on the new challenges and costs of managing a distributed system.

For horizontal scaling, we should look for CPU at 100% utilization, memory at the system limits, network throughput is at interface capacity, and Disk IO is a clear bottleneck. On top of that, check that none of those issues are due to the other issues mentioned in this guide. You may also see the need for ad hoc scaling to deal with incoming traffic waves such as "black friday sales" events for ecommerce, or new specialized workloads that require isolation such as running an LLM on your infrastructure. In general, if you aren't serving billions of daily requests seriously question if you have scale issues or performance issues.

"Kafka is not fast enough for our 50k records"

A team of senior and staff engineers concluded that Kafka was unable to deliver more than a few hundred messages per second, despite Kafka being well-known to handle millions. The reasoning was a result of their Python implementation which took hours to process thousands of messages with excessive failures and lost messages.

The implementation created new connections (to the Schema Registry and the Kafka Topic) for every message:

    for message in messages:
        # Creating new SchemaRegistryClient, AvroSerializer, and others for EVERY message!
        # Consumes sockets and memory, spams infrastructure with connections.
        registry = SchemaRegistryClient({'url': 'http://localhost:8081'})
        schema = registry.get_latest_version('events-value').schema.schema_str
        serializer = AvroSerializer(registry, schema)
        producer = Producer({'bootstrap.servers': 'localhost:9092'})

        serialized = serializer(message, None)
        producer.produce('events', value=serialized)
        producer.flush()
        # No cleanup

The team planned to implement their own Kafka client which would have been very time consuming, risky, and expensive (up-front and on-going). The actual fix was moving initialization outside the loop:

    registry = SchemaRegistryClient({'url': 'http://localhost:8081'})
    schema = registry.get_latest_version('events-value').schema.schema_str
    serializer = AvroSerializer(registry, schema)
    producer = Producer({'bootstrap.servers': 'localhost:9092'})

    for message in messages:
      producer.produce('events', value=serialized, callback=delivery_report)

    producer.flush() # Even this part can be improved further.

That change sent thousands of messages nearly instantly with no failures. Previously it took hours with a high failure rate.

This is the point of knowing when your teams mis-diagnose socket exhaustion. One developer moving a few lines of code outside of the loop, or your most senior talent spending weeks or months re-inventing Kafka packages, and maintaining those packages forever.

Why other solutions fall short

When socket exhaustion crashes your service, pressure mounts to "fix it!"

Teams often reach for familiar and popular scaling patterns or architectural changes. Those approaches fail when socket exhaustion is the root cause, wasting time and money.

Marginally helpful but still wrong:

• Add more servers and task queues - Spreads the problem across more machines, requires rewriting logic to correctly re-fetch context (risky). Each server still crashes when its socket limit hits. Expensive.
• Increase ulimit. Buys time but doesn't fix the leak. Eventually you hit kernel memory limits anyway.
• Switch to async/await. Async doesn't reduce socket count if you're creating new connections. It only makes them "non-blocking" from the perspective of how many you can create at once in the language.
• Move to microservices. Depending on deployment strategy this can add a "server failure bulkhead" to prevent company-wide system failure. However, whatever service is exhausting sockets will still continue to exhaust sockets.
• Add Redis cache - Caching API responses doesn't reduce socket creation if the client pattern is wrong. It may reduce the number of requests your code makes depending on your use-case.

Not helpful:

• Rewrite in Go/Rust for "better performance" - Language doesn't fix client initialization in loops. Same bug, new codebase. However, you might accidentally fix the real issue in a new implementation.

All these options delay resolution, increase complexity, and cost more than moving 3 lines of code around.

Organizational audit checklist

• Review all bulk processing code (emails, webhooks, data sync)
• Audit API client initialization patterns across codebase
• Set up socket usage monitoring/alerting
• Document connection pooling standards for the team
• Recovery happens after waiting or restart