Memory Sawtooths

Context: The server regularly hits a memory limits for recurring processing work, and it seems to be getting worse and more frequent.

Common application behaviour

When a memory sawtooth hits, it is typically silent and unnoticed until the last moment.

Hard to see it from a data dashboard as the system memory is "fuzzy", mixed in with other unrelated workloads using memory.

What's going on

Commonly we see software systems crash due to running out of memory.

Typically that is due to some data-transformation process that happens on the same server. Many people reach for major distributed architectures to solve this but you would be amazed at what a simple single system can manage.

(Diagram of memory sawtooth growing and growing, eventually hitting 100% and the system crashing.)

A distributed system does not address the heart of the issue and assumes a priori that it must be processed the way it is.

We commonly see data transformation code like this

function transform(raw_data)

processed_step_1 = step1(raw_data);
processed_step_2 = step1(processed_step_1);
processed_step_3 = step1(processed_step_3);

return processed_step_3
}

function step1{
  ...
}

todo make this code more realistic.

Perhaps your steps are not broken into clean functions Perhaps your function steps are sort of scatted across the code base, some parts owned by other teams, some parts too complex to change.

For example, this is a common issue in many MVC web frameworks. They often have an ORM or database layer which can query many records as raw data, the internals often transform that to some object representation by passing the full array of raw data to a transformer which returns an array of objects

//clean up all this code to make it more believeable
function queryLogs() {
  rawLogs = getLogsFromDB() // return 100k records [...] sawtooth 1
  logs = []
  for logItem in rawLogs //sawtooth 2
    logs.push(new Log(logItem));

  return logs;
}

function renderLogs(logs) {
  responseHTML = "";
  for log in logs //sawtooth 3, big string
    responseHTML += "someHTML" + log.data + "more html"

  httpd.send(responseHTML)
}

For anyone making very large reporting or data pages, you have probably run into an out-of-memory style error at least once in one of these layers when doing this work.

The really tragic part of this is that depending on your language you might not be told "out of memory" but rather something entirely unrelated, and just linked to what logical bits of the code stopped working elsewhere when memory ran out. For example, I've seen systems claim the DB is unavailable simply because memory peaked.

The general issue is that this architecture insists each step must complete an entire logical batch of data before sending it off, and that the batch size is equal to the entire data set.

How to fix it

// todo fix this code example so it shows generators at play
function transform(raw_data)

processed_step_1 = await step1(raw_data);
processed_step_2 = await step1(processed_step_1);
processed_step_3 = await step1(processed_step_3);

return processed_step_3
}

Simply by using a generator we can flow data record-by-record through the steps, preventing massive build-ups of memory sawtooth patterns entirely.

There are times where you do require the entire group of data to do some operations. (need to figure out what to say about that)

Real mistakes I've seen

thinI need examples, but they are kind of common perhaps?o

Memory Sawtooths