Memory Tools: Finding Out What Your Program Is Holding

A no-GC language makes memory questions answerable in a way a garbage-collected one cannot — the arena either gave the bytes back or it did not, and there is no collector in the middle to argue with. This page is the set of instruments that answer, in the order you should reach for them.

If you want the model — arenas, local, scoping rules — read Memory Management Best Practices first. If you want the event log specifically, Memory-Event Log is its full guide. This page is about diagnosis: something is growing, what is it and why.

The distinction that decides everything

SuperJ has no garbage collector and the global arena never frees. So there are two completely different ways for a process to grow, they look identical from the outside, and they have opposite fixes:

1. A retention leak. Something keeps a reference: a cache that never evicts, a list that only appends, a static that accumulates. The memory is reachable. Eventually you run out. This is the leak you already know from Java.

2. Garbage with no scope. Nothing points at the memory at all — it became unreachable the moment the work finished — but nothing frees it either, because it was allocated in the global arena and the global arena is process-lifetime. A garbage collector would reclaim this; SuperJ will not.

The second kind does not exist in Java, so it is the one people miss. It is also, in a server, usually the bigger half. A request or a connection allocates temporaries; if no scope ends when the request does, every one of those temporaries is permanent.

System.memAudit() is the only tool that tells the two apart. Everything else tells you what allocated; this tells you why it is still there.

Step 1 — Is it actually growing?

warmUp();                                   // let one-time costs happen first
long before = System.memTouchedBytes();
for (int i = 0; i < 500; i++) {
    doOneRequest();
}
long grew = System.memTouchedBytes() - before;
System.out.println("grew " + grew + " bytes over 500 requests");

memTouchedBytes() is byte-exact and costs nothing — it is live on every build, not just debug ones.

Use memTouchedBytes(), not memReservedBytes(), for this. Reserved bytes count arena chunk reservation, so the number moves in chunk-sized steps. A small per-request leak is invisible inside the current chunk's unused slack — and because chunks double in size, that blind spot grows the longer the process runs. A measured 1 KB-per-round leak sat at exactly its starting value for 965 rounds, nearly a megabyte of real leakage, before the counter moved at all. One of the SDK's own leaks regressed twice past tests that bounded reserved bytes.

>

Use memReservedBytes() for the other question — did that arena {} block give its memory back? — where the delta across a scoped call must be 0 and the chunk granularity does not matter.

>

Take that delta across the call, not inside the method doing the work. A method that drops a callee's fresh result gets an implicit frame region, and a reading taken inside it counts the region's live 256 B first chunk: a stable +256, once, that is reclaimed at return (#5163; see frame regions).

In tests, you get this for free:

superj test --check-growth

Every @Test runs twice. The first run absorbs the legitimate one-time costs — lazy statics, a first-use cache, a builder growing to its steady size — and the second must retain exactly zero bytes. A test whose whole point is holding memory can be exempted:

superj test --check-growth --allow-growth CacheTest

Step 2 — Is the growth linear?

Run the same workload at 100, 200 and 400 units and divide by the unit count. If the per-unit figure is constant, growth is linear and the process cannot run indefinitely — it will be OOM-killed, and the only question is when. If the per-unit figure falls, you are looking at amortised warm-up, which is fine.

This is worth doing explicitly, because "it uses a few KB per request" and "it will die after nine hours" are the same sentence, and only one of them makes people act.

Step 3 — What allocated it?

Rebuild with one flag:

superj compile app.sj --sdk-path $SJ_HOME/sdk --link --mem-debug --output app
./app

At exit the program prints two tables: one per type, one per source line.

The per-line table is usually the one you want, because it distinguishes a constructor's new byte[256] from the new Session() that calls it — the per-type table would report both as Session and <raw/array> and leave you guessing.

For a rate rather than a total, bracket the suspect work with reports:

warmUp();
System.memReport("baseline");
for (int i = 0; i < 100000; i++) {
    handle(i);
}
System.memReport("after 100k");

Each report re-baselines, so consecutive calls read as "what grew between these two points". Once a baseline exists the table is ordered by growth rather than by total — which matters more than it sounds: a 200 MB lookup table loaded once at startup outranks the sixteen bytes a request retains in every total-ordered view, forever.

For a long-running process you do not want to edit, set SJ_MEM_HIST_EVERY=<bytes> and the runtime emits a report every N bytes of global allocation on its own.

Read global_bytes as the leak column — the global arena never gives memory back, so its cumulative bytes are its live bytes.

The decision column: why it is there, not just where

The last column says what the compiler decided about the allocation, which is the half a byte count cannot tell you. Three of its values are all global traffic and they are three different problems with three different fixes:

A starred value like GLOBAL* means the run resolved it rather than the compiler: the sj_galloc family chooses its region at run time from whatever is armed, so the honest compile-time answer there is "ambient", and the star marks the difference between a decision and an observation.

PROMOTED is the one that changes what you do. It tells you the allocation is fine and the store is the problem — which is not visible at the line the bytes are attributed to. block_bytes is churn that an arena {} block has already reclaimed, which is a block doing its job.

Check the coverage: line at the bottom of the table before trusting it. Site ids are emitted by the calling code, so an allocation made inside the prebuilt SDK archive emits no site at all and cannot even show up as unattributed. String bodies are the main uncovered kind.

>

If coverage is low, rebuild with --sdk-source instead of --sdk-path. That compiles the SDK with the same flag, so its own lines carry site ids and the table attributes them. This is the difference between a dead end and an answer when the bytes are arrays: the per-type table calls every array <raw/array>, which names nothing. On one investigation (#4967) the archive build reported coverage: 320 of 112760 retained bytes attributed to a source line (0%) and the per-type table showed <raw/array> 104800 — the source build attributed all of it to four lines in three SDK files, in one run. Expect the build to take minutes rather than seconds.

The carved-calls section: which call cost the block its reclaim

CARVE-OUT is in the table above for completeness, but you will almost never see it there, and the reason is worth knowing because it points at the section that does answer the question. A carve-out is a property of a call, not of an allocation: the compiler disarms the ambient region across a call whose callee cannot promise it keeps nothing it allocates, so the bytes land in the callee, several frames away from the line you would have to change.

So the report has a third section (#4874), printed after the per-line table:

== carved calls (debug build, SJ_MEM_DEBUG) ==
  a call inside an `arena {}` block whose callee cannot promise it keeps nothing
  it allocates is CARVED OUT (#3805 C1): the ambient region is the global arena
  across it, so what the callee allocates outlives the block. These are the bytes
  the block did not reclaim, charged to the CALL — which is where the fix is.
  carved call                    global_bytes   glob_cnt      calls   per_call
  Session.sj:88                      14400000     100000     100000        144

Read it as: this line is why your arena { } block is not reclaiming what you expected. per_call is the price of one call, which is the number to compare against the size of the block you thought you were scoping.

Three fixes, in the order worth trying:

1. Summarize the callee. The carve-out exists because the callee could not be proven to keep nothing it allocates. If it genuinely keeps nothing, saying so removes the carve-out and the bytes move into the block. W_ARENA_CARVE_OUT (a warning inside strict arena) names the same calls at compile time — it is the same fact, before the run.
2. Move the call out of the block, if what it allocates was always meant to outlive it. Then nothing changes except that the report stops implying otherwise.
3. Give the work its own region (doc/regions.md), when the lifetime is not lexical.

These are the same bytes the per-type table counts, and — where the callee was built with --mem-hist — the same ones the per-line table shows under the callee's own lines. Three views of one set of bytes. Do not add the sections together.

The section covers allocations made inside the prebuilt SDK archive too, which the per-line table above cannot: it is charged at the allocator rather than from a compile-time site id, and an unsummarizable SDK method is the commonest callee a carve-out exists for.

Step 4 — Why is it still there?

This is the step that decides the fix, and it is the one most often skipped — the per-line table looks like an answer, and it is only half of one.

System.memAudit("after 200 connections");

memAudit traces the global arena from the program's roots and reports every allocation that nothing points at any more, grouped by the source line that made it and by the store that orphaned it, with a reachable: total at the end.

Here it is on a cache of exactly 100 keys, after 5,000 requests that each put a fresh 64-byte value:

== garbage audit — after 5000 ==
  unreachable allocations in the global arena: nothing points at these, and
  an arena never frees one, so they are held until the process exits.
  site                                      garbage_bytes      count  orphaned at
  <site:0>                                         226279      15929  <unknown>
  Leak.sj:4                                        367200       5100  Leak.sj:4
  TOTAL GARBAGE                                    593479      21029
  reachable: 14542 bytes in 303 allocations
  Conservative: an integer that looks like a heap address keeps an object
  alive, so this UNDER-reports. Everything named above is really unreachable.
== end garbage audit ==

The reachable 14 KB is the cache itself — bounded, correct, exactly what was intended. The 593 KB of garbage is every value the cache overwrote. No compiler warning fires, the live set never grows, and the process still dies.

orphaned at: the line to change

Knowing where the bytes came from is not the same as knowing what to edit. For a value that is replaced, the line that allocated it and the line that dropped the last reference to it are different lines, and only the second one can be changed. The runtime does not have to work this out — it watched the store happen — so the audit reports it.

Two things about that column are worth knowing:

• It carries across ownership. A buffer allocated inside a constructor became garbage because its owner was replaced; nothing ever stored over the buffer itself. The audit propagates the owner's drop site to it, so instead of one unexplained row for "the constructor's buffer" you get one row per store that orphaned a batch of them. One allocation site reached from three stores is three rows, because it is three leaks with three fixes.
• <unknown> is honest, not missing. It means no instrumented store dropped it. The usual cause is that the drop happened inside the prebuilt SDK archive — cache.put(k, v) replaces the old value with a store inside Obj2ObjMap, which is compiled without --mem-hist and emits no site. Building with --sdk-source makes the SDK's own stores visible, and that case then reports the exact line inside the container. The audit will not guess the caller.

Run it at two or three scales and watch the reachable: line. That one number is the whole diagnosis:

For the second row, the fix is scoping, not pooling:

• If the lifetime is a block — a batch, a parse, a loop body — wrap it in arena { }.
• If the lifetime is not a block — a connection spanning many event callbacks, a request spanning a parse, a dispatch and a write — use a region: Region.open() when the work starts, arena (r) { … } around each callback, r.close() when it ends. See ../doc/regions.md and SPEC §7.10.
• Pooling individual objects also works, but it cannot reach zero: you can only pool an object you can name and hold, and the temporaries — a scratch buffer, a parse, a copy — have nothing to be pooled into.

The example above is worth dwelling on, because it is a leak class no compiler check can ever reach. A bounded cache whose values are replaced holds a perfectly constant live set and grows the arena forever, and the reason no claim catches it is structural: escape analysis asks where an allocation may travel, and here every allocation travels exactly where it is meant to. What leaks is the value that was overwritten — a property of the assignment, not of any allocation site. (A 1,000-key version of the same shape measured 176 bytes retained per request, with no diagnostic from anything.)

It reports; it never reclaims. A collector would make that code survivable — the point here is to make it fixable.

memAudit is deliberately conservative: any aligned word that happens to look like a heap address keeps what it points at alive. So it under-reports. Everything it names is genuinely unreachable, and its failure mode is staying quiet about a leak, never accusing you of one that is not there.

Step 5 — Ask the compiler (you can do this first)

Several of these are free and run at compile time. If one of them fires, you can skip the whole measurement loop.

Three warnings are on by default, because nobody would think to ask for them:

• W_ARENA_NO_SAVINGS — an arena { } block whose every allocation is promoted out, so it reclaims nothing at all.
• W_ARENA_CARVE_OUT (inside strict arena) — a call the block cannot prove safe, so its allocations went to the global arena anyway and the block collected less than it looks like.
• W_ALLOC_PER_CALL_IN_LOOP — a call in a loop to a method whose author marked it @AllocatesPerCall: it allocates on every call, and what it allocates reaches the global arena, which never frees. The point of this one is where it appears. The allocation happens inside the callee — very often inside the prebuilt SDK archive, which emits no site id — so neither the per-line memory table nor --warn-alloc-in-loop can see it, and nothing at the loop said a word. When the SDK's caller-buffer sibling exists (listInto for list), the message names it. Default-on, suppressible with @NoWarn. The marked set is deliberately small: it is an author's claim, not an inference, because "returns something freshly allocated" is true of toString() and every builder and a warning on that is a warning everyone turns off.
• W_CTOR_BUFFER_IN_LOOP — a loop-local wrapper the analysis did stack-promote, whose constructor allocates a buffer into this anyway. for (…) { ByteArray b = new ByteArray(64); } costs nothing for the wrapper and 64 bytes per iteration for the array, and the call site shows neither.

And you can make the compiler enforce intent rather than guess it: strict arena n { }, local, and @NoRetain turn "probably fine" into a compile error when it is not. See The Claim Layer.

Step 6 — When you suspect a use-after-drop

Reading arena memory after its block was dropped is not a crash — free() leaves the bytes readable, so the program keeps running with stale data and only fails once an intervening allocation reuses the chunk. Small inputs pass; a big one crashes somewhere unrelated.

--mem-debug fills dropped chunks with 0xDE, which turns that into an immediate, unmistakable failure:

Fatal: SIGSEGV at address 0xdededededededede

If you are doing anything with manual lifetimes — regions, pooled objects, a native that holds a pointer — run your test suite once under --mem-debug before believing it works.

--ea-check catches the mistake at the store, not the read. --mem-debug shows a use-after-free only when the freed memory is read after being reused. --ea-check includes everything --mem-debug does, and also checks every pointer store, return and block exit against the lifetimes of the regions involved. The first store that puts a pointer where its region will be freed first stops the program with exit code 86, naming the store's line and where the region was opened:

superj compile app.sj --sdk-path $SJ_HOME/sdk --link --ea-check --output app && ./app
EA-CHECK: store at app.sj:41 puts a pointer into arena block #7 (opened at app.sj:38) where the
global arena holds it -- the first is freed while the second can still reach it (use-after-free)

It works with superj build, run and test too. Check builds are slower (every function stays out of line), so use them for test runs, not production. Stores made inside the prebuilt SDK are not checked unless you compile with --sdk-source. Set SJ_EA_CHECK_SUMMARY=1 to see how many stores were checked.

A worked example

The SDK's own TLS server path, tracked through a memory campaign:

Good progress — and still linear, so still an eventual OOM. The per-site table said which lines allocated, and each round of pooling removed one. It never reached zero, and it was not obvious why.

memAudit, at three scales, in the fully pooled configuration:

Reachable exactly constant — the same bytes in the same allocations at every scale. Nothing was retaining anything; there was no reference leak and there never had been. Every byte of growth was unreachable the instant the connection closed. The pooling had been attacking the wrong problem all along, and could never have finished: what was left were transients with nothing to be pooled into.

The actual cause was that a connection has no scope. arena { } is lexical; a connection spans many poll() callbacks with no block around it. Giving it one — a region — took the same path to 1,008 bytes per connection with all pooling switched off, and the reachable set from 509,482 bytes to 823.

One memAudit call answered a question that weeks of per-site tables had not.

Quick reference

memReport and memAudit are no-ops on a non-debug build, so calls to them can stay in source that ships.

See also

• Memory Management Best Practices — the allocation ladder, scratch buffers, and how to write code that does not need this page
• Memory-Event Log — the full guide to --mem-track / superj memlog
• Memory Safety — what the language guarantees
• The Claim Layer — strict arena, local, @NoRetain
• ../doc/regions.md — scoping a lifetime that is not a block