Performance, Not Scale: Four Problems to Check First
- No sales pitch here, just hard-earned lessons.
Performance issues are frequently misdiagnosed as scaling problems. This critical mistake leads organizations to implement expensive, complex solutions (more servers, distributed architectures, etc.) when simpler, targeted fixes would solve the underlying problems more effectively.
This pattern doesn't just drain resources temporarily, it creates a vicious cycle that permanently suppresses your Revenue per Engineer (RpE) metrics and erodes profitability through:
Performance, Not Scale: Four Problems to Check First
The financial impact of this misdiagnosis goes beyond immediate project costs. It creates a self-reinforcing cycle that degrades your engineering economics:
Performance issues are frequently mistaken for things that need scaling solutions (such as more servers, bigger servers, complex architectures, distributed patterns, etc). This means taking something that doesn't work well, and wrapping it in a high-complexity, high-cost solutions to prop-up the problem a bit longer. While that might unblock revenue stoppage by preventing the system from falling over for the next few quarters or even years, they end up being very expensive, especially when simpler and more cost-effective solutions can solve the core problem.
Scaling poor performance is the missteps that drives down Revenue per Engineer and suppresses profitability of your business on an on-going basis through these consequences:
- Scaling poor performance pushes your engineering mind-share and time-share disproportionately into complex solutions
- Increasing technical complexity which increases demand for headcount growth, which lowers RpE
- Increasing technical complexity lowers or stagnates product quality/differentiation, which later pushes out revenues, which lowers RpE
- Increasing technical complexity without proportional ROI establishes a vicious cycle of strategic dilution feeding tech degradation
The "big four" issues that show up time and time again are:
Why these 4? First, they are frequently found when performance issues show up, and over-engineered in regard to the performance problems.
Scaling solutions themselves are not universally over-engineered or something to be avoided at all costs. They are the right choice for scaling problems. The issue is when scaling is misused to solve a performance issue.
Secondly, these "big four" are where companies most frequently burn cash pursuing scaling solutions that are both expensive and yield under-performing ROI. This additional complexity is undeserved at this stage, and acts as a sort of cognitive tax on future work which accelerates the costs. Further, as scaling solutions can only prop-up the underlying performance challenges, you are likely to hit the next catastrophe sooner than your peers who have solved the deeper performance challenges.
More fundamentally, all of these performance problems tend to be a side-effect of something deeper. Culture, tooling, knowledge, process issues, poor software monitoring, poor release processes, poor standards. Even the mindset of a few seniors can allow "magical thinking" (guesses, rumors, myths, gut-feelings) to explain problems, avoiding clear and logical cuase-effect research).
These problems should be addressed in your company too, but will not be in this guide.
Subscribe to the mailing list below to know when those guides come out.
The Performance/Scale solution gap
Scaling a performance problem is like trying to heat an uninsulated house with all the windows open. Instead of addressing those first-level lower-cost issues we see businesses opt for installing complex industrial-grade furnaces to heat the house. Sure, it will be warm, and yes that's what the biggest of buildings use to heat their spaces, but a proper cost-effective solution must come in layers. Even those big spaces close the windows and have insulation in place, and that's why they can still out-scale you even when you use the same furnances they do.
The gap between problem and solution is staggering. Teams frequently propose 6-month to multi-year migrations to fix what's caused by a 1-line SQL statement. One of the most common patterns is reaching for horizontal auto-scaling and worker queues to manage just thousands or tens of thousands of API requests, when they really just need to move an API client initialization outside of a for-loop (as one of the sub-guides will show). The solutions can be as minimal as 10 lines of code, yet the default thing we find is a massive architectural overhaul (or sometimes things that sound impressive on a job applicant's profile).
Unfortunately, teams that have these issues frequently do not recognize them as these 4 issues, often due to a lack of familiarity (which this guide aims to alleviate).
With well over a decade of hands-on experience I have personally seen at-least 1 of these 4 issues in every single company I've looked at. That's dozens of companies between Europe and North America. Further, simply discussing these issues with hundreds of other engineers at organizations I have not been part of confirms they often see these issues in their own companies too.
Unfortunately, not only are the "big four" performance problems common, but the anti-pattern of trying to fix performance with scaling solutions is just as common too. There are always some active mega construction sites building out web-scale solutions for workloads that could easily fit into a handful of computers (or even one computer). It happens in a two-person startup, 1000 person tech company, and 10k person enterprise.
Why These Issues Are So Costly
The "big four" performance problems are particularly dangerous because they:
- Hide until it's too late - They appear normal in development, testing, and possibly fine for years untouched in production.
- "Scale" with growth - Problems increase as your business grows, not always linearly or predictably.
- Encourages overengineering - Slow-building consensus that scaling solutions are needed despite being disproportionate to the actual problem.
- Undermine confidence - Recurring performance problems erode business trust in engineering's ability to support the business.
The cost differential between targeted performance fixes and scaling solutions is staggering: typically €1K and a few developer-days versus €2M and multiple teams over quarters.
Why These Issues Exist
They are camouflaged. In your developer's environment, in testing, and in production.
A normal feature gets built and released. Nobody changes it, but somehow a few reports of occasional slowness or breaking pop up. With time, those reports become random crashes, and sometimes they bring down important workflows or the entire website. These issues generate constant business escalations and can even generate financial loss. As this continues, the rest of your business begins to lose faith in engineering's ability to build and run reliable systems they can depend on.
That creates major pressure to make drastic changes like a full rewrite, changing out tech leadership, re-structing teams, out-sourcing development to reduce costs, placing tighter micromanagement processes in place, or even the business strategy itself becoming so constrained by technology it constantly compromises down to a less impactful roadmap
Tech orgs often fall into the same predictable scaling performance problems traps. It can look like pushing for microservices, more powerful server instances, Kafka, Kubernetes, message queues, events, CQRS, orchestration platforms, and complete rewrites. All of those are wonderful solutions to problems that are not the "big-four" performance problems.
The thing is, this problem often happens without any direct code change. Sometimes the code might be "old code" but programs don't get Alzheimer's as they age. It's the same thing working the same way it did before.
And that's how it gets you. These issues always seem perfectly fine at first when introduced to the software. It is only as data usage patterns change they reveal the issues, or indirect code changes add use-cases that break the boundaries in ways that were never expected before.
These performance challenges have an increasing degradation pattern which grows until traumatic collapse of the system. That pattern of "slowly getting slower as we get bigger" tricks everyone into thinking slow is related to big, and they determine they hit the limits, so they look for scaling solutions to deal with what they thought were scale problems, and not identify it as a performance issue on the code level.
The Strategic Trap
The most unexpected part of these performance issues is how they reshape conversations for your technical strategy. As systems become unreliable, organizations often:
- Limit feature scope to avoid touching problematic systems
- Divert resources to firefighting and away from product differentiation
- Make increasingly conservative architectural decisions
- Increase process overhead in an attempt to prevent failures
This "play defensive" strategy creates what technology leaders try to avoid: a business that cannot grow fast supported by technology.
The Big Four
Missing database indexing
Query time grows exponentially with data size (O(n) vs O(log n)). A query that takes 10ms with 1,000 records takes 10 seconds with 1,000,000 records. Further, databases optimize results and might return repeated queried data faster, so if you run a query multiple times (just refreshing the page a few times) the hidden optimizer in the database might trick you into thinking results are coming back faster or there was some cache issue.
📄️ Missing Database Indexes
Your queries slow down exponentially as data grows. 10ms becomes 10 seconds. Find and fix the missing indexes.
Socket Exhaustion
Everything works perfectly until a socket limit hits, then everything crashes. Not just the part of the code making the request, but new requests from users trying to access your website are not available, new database connections, telemetry reporting tools, new connections to redis, really anything that needs a socket ends up broken. If you are lucky, some "cleanup" activity from the language or system tools might close the sockets for you, allowing the other parts of the service to come back online. It is not always obvious that a website suddenly not responding is due to socket exhaustion, especially as web servers are behind yet another HTTPD proxy server such as nginx which could be configured to abandon the request and return a 400 or 500 series error, fail-over to another working server, or even act as a request queue until the server comes back online. This error does not just impact the web app, it impacts the entire host system that ran out of socket resources.
📄️ Socket Exhaustion
Everything works fine until suddenly nothing does. Your entire system crashes when sockets run out.
N+1 queries
One logical query spams many sub-queries, possibly in an unpredictable way. In the first case, you may have a query that operates as a loop over an array, running the query many times. Alternatively, you need the results of the latest query to know if you need to make another query. The result is the same, one query can cascade into thousands if not careful. This problem is only seen when the correct mixture of values in the data are present. The kind of thing that could easily be overlooked when creating a feature with a narrow use-case in mind, and not thinking about the record with thousands of associated entities but tested and developed with 10, or a data structure that operates similar to a linked-list as far as lookups are concerned.
Memory Sawtooths
Processing data creates Growing waves of slowly-growing memory spikes until the uncleaned garbage raises the baseline too high, or a single task just has one-too-many records to fit into RAM. The result is the same: the system runs out of resources and crashes. Teams reach for bigger hardware with more ram to deal with the spikes instead of smoothing them out, or opt to implement some sort of large "background worker" process, especially attractive to teams are the more complex workers that involve a queue service and distribute messages to multiple workers.
Leadership In Action
Technology leaders keen to keep RpE high should ensure their teams are able to correctly identify and diagnose these big four problems whether in code, logs and metrics, runtime behaviour, or user complaints.
- Diagnostic discipline - Establish proper performance analysis as a requirement before approving scaling solutions
- Engineering culture shift - Reward thorough problem diagnosis over complex solution implementation
- Technical standards - Create guardrails that prevent these issues through code reviews and automated testing
- Knowledge investment - Ensure your teams recognize these patterns through targeted training
Most importantly, these solutions require executive sponsorship to prioritize attention to seemingly "basic" code-level work instead of kicking off major projects around hype-friendly architectural solutions. Things like K8s, Kafka and Microservices are heavily discussed tech news and become a superficial "bar" for companies to meet if they want to prove what they do is "hard", as if keeping tech complexity low is a source of shame in their peer groups and and a blief it brings poor positioning at the next fund-raising round.
The companies that maximize RpE aren't those with the most cutting-edge and complex architectures, they're those who understand when sophistication is necessary and when it's wasteful, and trying to scale your way out of performance problems is massively wasteful.