open https://gitlab.synchro.net/main/sbbs/-/issues/1185

## Summary

`sbbsctrl.exe` (Windows, 32-bit, hosting all servers in-process) crashed with an
access violation deep in the SpiderMonkey GC/mark path while a **web SSJS request** was executing JavaScript. The web server (and with it the whole BBS) went offline until manually restarted ~8 hours later.

This is **not** the previously-fixed `js_rtpool` NULL-deref
(`fab8b1f1d`, which faulted in `JS_TriggerAllOperationCallbacks` after an OOM) —
different signature, and there was **no OOM** preceding this one.

## Environment

- Host: vert.synchro.net, Windows 11, **32-bit** `sbbsctrl.exe` (all servers
in one process), SpiderMonkey 1.8.5 (`mozjs185-1.0.dll`).
- Synchronet 3.22a, `master`, Debug build (the build carrying the web/terminal
teardown fix — the crash is unrelated to that change; see "Ruled out").
- Crash: 2026-07-14 ~02:13 local. WER minidump
`sbbsctrl.exe.233816.dmp` (preserved off-box).

## Fault

Access violation (read), `c0000005`, in `mozjs185-1.0.dll` GC/mark region (nearest exports: between `JS_CompareValues+…` and `js_CloseIterator`).

```
sbbs!js_execfile+0x1099
mozjs185_1_0!JS_ExecuteScript+0x22
mozjs185_1_0!<gc/mark, private> <-- faults here
```

Faulting instructions:

```
mov ecx, dword ptr [eax+0CCh] ; eax = 0x16d45a5a (a JS object), loads ecx
mov eax, dword ptr [ecx+18h] ; ecx = 0xdf6807ff (WILD pointer) -> AV read @ 0xdf680817
```

So a GC object carried a **garbage pointer (`0xdf6807ff`) in a field at `+0xCC`**, and the collector dereferenced it. Classic
use-after-free / heap-corruption manifestation: the GC is the *victim* tripping over corruption some earlier code left behind.

## Context at crash

- Under an ongoing distributed L7 flood; server busy but **not** saturated
(~54 concurrent clients, not at the 100 MaxClients cap).
- Dozens of concurrent HTTPS `<Guest>` sessions all executing the **same
script**, `001-forum.ssjs` (the message-board viewer:
`GET /?page=001-forum.ssjs&sub=…&thread=…`), at the moment of the fault. - **No OOM** — zero "creating JavaScript runtime" / "Failed to create new
context" / "out of memory" errors in the hour before the crash.

Leading (unproven) hypothesis: a GC/rooting or JS-runtime-lifetime issue exposed by **high-concurrency SSJS execution** of the forum page — consistent with "many threads in the same script, no OOM," but not established.

## Ruled out

- **Not** the connection-teardown fix (that touches `close_session_socket` /
node teardown; this is request *execution*, a separate path).
- **Not** the `js_rtpool` OOM NULL-deref (`fab8b1f1d`) — different faulting
function, and no OOM here.

## RCA status: characterized, not root-caused

The WER dump is a **minidump** (flags: `DataSegs | ProcessThreadData`, ~29 MB of
a ~655 MB process) — it captured thread stacks but **not the JS heap**. Every heap address (the corrupted object, the wild pointer's target) reads `????????`, so the offending object can't be inspected. And the code that *wrote* the bad pointer had already returned before the GC tripped over it, so it is on no captured stack. This dump therefore cannot yield the culprit.

## Next steps to get a real RCA

1. **Full-memory dumps now enabled** for `sbbsctrl.exe` via WER `LocalDumps`
(`DumpType=2`) — the next occurrence will include the JS heap, letting us
identify the corrupted object's JSClass (→ which binding/script).
2. If a full dump still isn't conclusive (likely, since the writer has
returned): reproduce under a **page-heap** (`gflags /p /enable sbbsctrl.exe`)
or ASAN build, which traps the bad *write* at the moment it happens.
3. Open question worth recording: how much of this class the **SpiderMonkey 128
migration** is expected to eliminate vs. needing a fix in our bindings /
concurrency handling now.

Filing to track; will attach a full-memory dump if/when the crash recurs.

— *Authored by Claude (Claude Code), on behalf of @rswindell*
--- SBBSecho 3.37-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

https://gitlab.synchro.net/main/sbbs/-/issues/1185#note_9585

## Second occurrence — 2026-07-16 02:27 (with a FULL-memory dump this time)

WER `LocalDumps` `DumpType=2` was enabled after the first report, so this recurrence produced a **733 MB full-memory dump** (`sbbsctrl.exe.237928.dmp`, preserved locally; too large to attach here). Same fault as before:
`c0000005` read AV in the SpiderMonkey GC/mark path (`JS_CompareValues+0x2b6`, identical offset) under `sbbs!js_execfile → JS_ExecuteScript`.

### Confirmed: same script
The faulting thread's stack contains `001-forum.ssjs`, `index.xjs`, and `xjs_handler.js` — the **web forum-viewer** page (`GET /?page=001-forum.ssjs& sub=…&thread=…`), exactly the workload of the first crash. Driven by the ongoing distributed web-scraping flood.

### The traced value is not a valid object (type confusion)
```
mov ecx, [eax+0CCh] ; eax = 0x0e5a4343 (MISALIGNED — low bits 0x3 set)
mov eax, [ecx+18h] ; ecx = 0 → read [0x18] → AV
```
GC things are always aligned, so `eax` (`…4343`) is **not a real object pointer** — the collector is tracing a **mistagged / non-object jsval as if it
were an object** (`& ~7` → `0x0e5a4340`). `[eax+0xCC]` came back NULL, and the
next deref faulted. (In the first occurrence that same slot held a *wild* pointer, `0xdf6807ff`; here it's NULL — both are "marking something that isn't
a valid object.") Signature = **type confusion / heap corruption surfacing at GC time**, not an OOM.

### Not a concurrency race
Of 152 threads at the crash, exactly **1** was in JS/GC; 67 were web threads, the rest idle in socket I/O. So this is **not** a runtime-pool (`js_rtpool`) sharing/concurrent-GC race — it points to a single-thread corrupt/mistyped jsval in the forum.ssjs execution path.

### Binding audit (first pass) — narrowed, no definitive culprit yet `001-forum.ssjs` uniquely renders **poll messages**, enumerating lazily-built header fields: `header.poll_comments`, `header.poll_answers`, `header.tally`, and `field_list`. I reviewed their C++ builders in `js_msgbase.cpp`:
- `tally` (~2030): `UINT_TO_JSVAL` integers only — safe.
- `field_list` (~1541) and the msg-hdr string-list (~2073): array rooted before
the fill loop, strings null-checked, intermediates covered by SpiderMonkey's
newborn-root slots — no obvious unrooted-intermediate bug on inspection.

So a quick read did **not** find an obvious mistyped/unrooted jsval. The suspect area remains the rendering of attacker-influenceable message data (polls/votes — note the recently-fixed senderless-QWK-vote data class) in the forum viewer, but the dump shows the *victim* (the GC tripping over the bad value), not the *writer* (already returned).

### The specific msgbase: `syncprog` (both crashes) — and it has votes
The faulting thread's SMB file path in the dump is `…\subs\syncprog…`, and `sub=syncprog` appears throughout its memory (syncprog's cached headers show up ~19k times); the request was `?page=001-forum.ssjs&sub=syncprog&…`. This is the
**same sub as the first crash** (whose last logged request was `sub=syncprog&thread=37528`). Not every sub carries votes, so this is a useful discriminator — and syncprog *does*: an `smbutil l` of a copy shows real vote records (`*VOTE … on msg/poll #…`, e.g. #47786/#47878/#49235). So the crashing
workload is specifically the forum viewer **rendering a vote/poll-bearing sub**,
consistent with the poll/vote-field binding path above.

### Next step
A **page-heap / ASAN repro driving forum poll rendering** is the way to catch the offending write at the moment it happens — the full dump + code audit can't
pin the writer alone. Full dumps stay armed for further occurrences.

— *Authored by Claude (Claude Code), on behalf of @rswindell*
--- SBBSecho 3.37-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

https://gitlab.synchro.net/main/sbbs/-/issues/1185#note_9586

## Repro attempt (jsexec harness) — negative so far, and what it tells us

Built a jsexec harness that drives the exact bindings `001-forum.ssjs` uses against **syncprog** (`get_all_msg_headers` with/without votes, touching `poll_answers`/`poll_comments`/`tally`/`field_list`/`votes`), with `js.gc(true)`
forced repeatedly. Ran it under cdb (to catch any AV) at reduced JS heaps:

- **`-m16M`, 500 passes → DONE, no crash**
- **`-m8M`, 500 passes → DONE, no crash**

### Why the single-process harness doesn't reproduce it
The full dump showed only **one** runtime/thread in GC at the crash, so the bad jsval lives in a single runtime's heap — which is why a single-runtime jsexec *should* reproduce it if the trigger were per-runtime GC timing. It doesn't. Combined with the two field observations — **32-bit only** and **only under scrape-level pressure** — the trigger looks like **process-level address-space
pressure, not per-runtime GC pressure**:

- `-m8M` shrinks only the *runtime's* GC heap; the jsexec *process* still has a
clean, unfragmented ~2 GB. So forced GC fires, but never in the degraded
allocator state the bug needs.
- On the live 32-bit web server under a scrape, `js_rtpool` hands each of dozens
of concurrent request threads its own runtime (vert: `JavaScriptMaxBytes=64M`
each). Their sum approaches the 2 GB ceiling → the process address space is
**fragmented / near-exhausted**. In that state a JS/GC allocation takes a
different path (high/odd address, near-failure), which is a plausible origin
for the **misaligned/mistyped jsval** (`0x0e5a4343`) the GC then traces.

This fits every data point: 32-bit-only (address space is the 32-bit-specific resource), scrape-only (concurrency creates the aggregate pressure), and single-thread-in-GC (one runtime is the victim, but the whole process being squeezed is the *condition*).

### Next repro step
A **v3 harness** now tries to recreate the condition inside one jsexec: consume process address space with live ballast allocations to near-exhaustion, then render syncprog polls in the tight, fragmented remainder (hovering at the OOM edge). If that still comes up empty, the single-process route is likely a dead end and the definitive path is a **debug web server + gflags page-heap under concurrent `?page=001-forum.ssjs&sub=syncprog` load** — which recreates the exact
multi-runtime condition and traps the bad write in the act. Full-memory dumps stay armed for the next live occurrence either way.

— *Authored by Claude (Claude Code), on behalf of @rswindell*
--- SBBSecho 3.37-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

https://gitlab.synchro.net/main/sbbs/-/issues/1185#note_9592

## LAA (`/LARGEADDRESSAWARE`) as a candidate stopgap — partial safety test

Since the crash correlates with **2 GB address-space exhaustion** on a 32-bit, non-LAA process under a scrape (dozens of concurrent `js_rtpool` runtimes), a candidate stopgap (until the SM128/64-bit migration) is to link the host `/LARGEADDRESSAWARE` for a ~4 GB ceiling on 64-bit Windows. The risk: SM 1.8.5 and other components were never validated above 2 GB.

**MSVC/mozjs side — tested, looks OK.** Patched a copy of `jsexec.exe` LAA and
ran the syncprog poll-render harness with ~3.4 GB of ballast so allocations crossed 2 GB (Windows allocates bottom-up), forcing GC repeatedly. 300 passes, **no crash**. Because TraceMonkey is compiled in, the harness's own hot loops were trace-JIT'd and executed above 2 GB too. So the mozjs **GC core** (mark/ sweep, chunk lookup) and **JIT hot paths** tolerated high addresses in this build. Not a full acquittal (one workload; doesn't hit the full JIT/trace surface a real forum render would), but on-point evidence against the conservative-scan-UAF concern.

**The bigger, UNTESTED risk: the Borland host.** `jsexec`/`sbbs.dll`/`mozjs` are
all MSVC. The process that actually crashes is **`sbbsctrl.exe`** — the 2006-era
Borland C++ Builder VCL GUI host — whose **RTL / VCL memory manager** governs the process's heap and address-space behavior, *not* MSVC's. That runtime predates LAA and is exactly the kind of code that may use signed pointer math / high-bit assumptions, and it can't easily be fixed if it's broken. So LAA on the
GUI host is gated on whether the Borland runtime tolerates >2 GB — which the jsexec test did **not** cover.

**Decisive test (planned):** `editbin /LARGEADDRESSAWARE` a *test* copy of `sbbsctrl.exe`, set `AllocationPreference = MEM_TOP_DOWN` (system-wide, reboot) so its *first* Borland allocations land above 2 GB, and see whether it simply **launches and runs stably** — a startup-time answer, independent of load.

**Cleaner alternative:** LAA is far lower-risk on the **MSVC** hosts — `sbbscon.exe` / `sbbsNTsvcs` — which carry no Borland RTL. If a deployment can
run the console/service host instead of the GUI, LAA becomes an all-MSVC proposition (the regime the jsexec test already de-risks), sidestepping the Borland unknown entirely.

(Repro status unchanged: single-process jsexec — v2 tiny-heap, v3 2 GB, v3+LAA
4 GB — still does not reproduce the original crash, consistent with it needing
the web server's multi-runtime rtpool churn.)

— *Authored by Claude (Claude Code), on behalf of @rswindell*
--- SBBSecho 3.37-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

https://gitlab.synchro.net/main/sbbs/-/issues/1185#note_9593

## LAA Phase-1 gate: the Borland `sbbsctrl.exe` host **survives** LAA + top-down allocation

The open question from the previous note was whether the 2006-era Borland C++ Builder RTL/VCL memory manager — which governs the `sbbsctrl.exe` process heap,
*not* MSVC's — tolerates pointers above 2 GB. It does.

**Setup.** `ctrl\build.bat` now marks `sbbsctrl.exe` `/LARGEADDRESSAWARE` post-build (PE-header bit; Borland's ilink32 has no LAA option). System-wide `AllocationPreference = MEM_TOP_DOWN` (0x100000) was set and the host rebooted, so allocations come from the **top** of the address space — this front-loads high-address exposure to process startup rather than waiting for exhaustion.

**Result — clean:**

- **LAA confirmed active:** `!address -summary` reports 2.646 GB free = 66.14% of
total → a **~4 GB** address space (a non-LAA process would show ~2 GB).
- **The Borland host launched and runs stably:** 26 threads, all servers
listening (21/22/25/80/110/443).
- **Synchronet DLLs are executing above 2 GB** (relocated near the ceiling by
top-down): `SBBS` @ `0xfe1a0000`, `WEBSRVR` @ `0xfd250000`, `FTPSRVR` @
`0xfef80000`, `MAILSRVR` @ `0xff0f0000` — high bit set.
- **Serving normally**, and **zero errors** since start — no JS runtime/context
failures, no access violations.

(Note: `mozjs185-1.0.dll` itself loaded *low*, ~`0x6b750000` — DLL load addresses
follow ASLR, not the top-down preference. That's immaterial here: what matters for this bug is where the **GC heap objects** land, and those come from VirtualAlloc, which top-down drives to the top.)

**Taken with the earlier jsexec result** (mozjs GC core + trace-JIT hot loops clean with ~3.4 GB live above 2 GB, 300 passes), *both* halves of the process —
the **Borland host** and the **MSVC/mozjs** side — now have evidence of tolerating high addresses. The main risk that would have killed LAA as a stopgap
is not materialising.

### Not yet proven
This is startup + ~8 minutes of light traffic, **not** a scrape. Still open: sustained stability under real multi-runtime forum-scrape load (the workload that
actually triggers this bug), and whether a high-bit bug lurks in a path not yet exercised (JIT under real forum rendering, cryptlib under TLS load). Full-memory
dump capture stays armed, so a recurrence is analysable either way — and the faulting address distinguishes the cases: **>= 0x80000000** = a *new* LAA-introduced high-bit bug; the original low-address signature = LAA didn't help
(the bad jsval forms below 2 GB regardless) and is only a frequency reduction.

Also worth stating plainly: **top-down is a stress setting, not a production config** — it should be reverted once the Borland tolerance question is settled,
letting LAA run with normal bottom-up allocation. And LAA remains a **stopgap** that raises the ceiling; the real fix is the SM128 / 64-bit migration.

— *Authored by Claude (Claude Code), on behalf of @rswindell*
--- SBBSecho 3.37-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

https://gitlab.synchro.net/main/sbbs/-/issues/1185#note_9594

## Correction: deeper dump analysis disproves two earlier claims in this issue

Went back into the full-memory dump (`sbbsctrl.exe.237928.dmp`) and disassembled
around the fault properly. **Two things I asserted in the notes above are wrong**,
and should not be relied on. Correcting the record.

### ❌ WRONG: "access violation in the GC/mark path"
The stack is under **`JS_ExecuteScript`** — the **interpreter**, not a GC mark
phase. I inferred "GC" from the nearest *exported* symbol (`JS_CompareValues`), which is just the closest public name to an unnamed internal function — that was
guesswork stated with too much confidence. The function immediately preceding the
faulting one does **E4X** checks:

```
6dde5c43 cmp dword ptr [eax+4], 0FFFF0007h ; jsval tag == OBJECT?
6dde5c4a mov ecx, dword ptr [eax] ; ecx = payload (JSObject*)
6dde5c51 cmp dword ptr [ecx+4], offset mozjs185!js_XMLClass ; obj->clasp == XML?
```
So this is an **interpreter/value-operation** neighbourhood (E4X-aware), not GC tracing. Anyone chasing this should look at the interpreter path, not the collector.

### ❌ WRONG: "a tagged jsval being traced as a raw object pointer" `0xFFFF0007` is **`JSVAL_TAG_OBJECT`**, which proves this build uses the **nunbox32** value layout: a jsval is 8 bytes — `[+0]` raw payload, `[+4]` separate tag word. **There is no low-bit pointer tagging.** Therefore the misaligned `0x0e5a4343` is **plain pointer corruption**, not tag/type confusion.

### ❌ ALSO WRONG (a hypothesis formed and disproved during this pass)
I briefly thought the bug was the 3-byte misalignment (`0x0e5a4340` + 3, i.e. a corrupted low byte). It isn't: **`[0x0e5a4340 + 0xCC]` is `0x00000000` too** — at
the correctly *aligned* address the field is NULL as well, so it would fault identically. Misalignment is a symptom, not the cause.

### ✅ What actually holds up

**The faulting function is a small leaf accessor** (starts at `6dde5c70`; note the
`int 3` padding before it):
```
6dde5c70 mov ecx, [eax+0CCh] ; ecx = arg->[0xCC] <-- NOT null-checked
6dde5c76 mov eax, [ecx+18h] ; <-- FAULT (ecx == NULL)
6dde5c79 test eax,eax
6dde5c7b je 6dde5c84 ; (this one IS null-checked)
6dde5c7d mov eax, [eax+0D0h]
6dde5c83 ret
```
i.e. `return arg->[0xCC]->[0x18] ? ...->[0xD0] : 0;` — it guards `[ecx+0x18]` but
**not** `[eax+0xCC]`, so a bad `arg` becomes an AV instead of a clean bail.

**The bad pointer points into the interior of a JSContext.** `0x0e5a4343` sits in
committed private RW memory whose contents are unmistakably **`JSArenaPool`** records:
```
0e5a4344: first.next = 00000000
0e5a4348: first.base = 0e5a4358 \
0e5a434c: first.limit = 0e5a4358 > base==limit==avail -> empty arena 0e5a4350: first.avail = 0e5a4358 /
0e5a4354: current = 0e5a4344 -> points back at `first` (empty pool) 0e5a4358: arenasize = 00000fd8 (4056)
0e5a435c: mask = 00000007 (8-byte align)
```
(another pool follows: `arenasize=0xfd8`, `mask=3`). SM 1.8.5 embeds several `JSArenaPool`s in a **JSContext** (`stackPool`/`tempPool`/`regExpPool`), so the argument is pointing **into the middle of a JSContext**, not at any valid object
base — misaligned *and* interior. That is genuine pointer corruption.

**The caller is not identifiable from this dump.** The faulting function is a leaf
(no frame setup), so the unwind through it is unreliable — its apparent return
address `6dde78cf` actually follows a `call` to a *different* target (`6dde9690`),
i.e. that frame is stale stack data. The dump shows the **victim**; the code that
**wrote** the bad pointer had already returned.

### Net
Still **pointer corruption of unknown origin**, but better characterised: an **interpreter-path** (E4X-adjacent) accessor invoked with a pointer into a JSContext's interior. The "GC mark" framing in the earlier notes is a red herring
— disregard it. Pinning the writer still needs a page-heap/ASAN repro or a lucky
dump that catches the corrupting thread in the act.

— *Authored by Claude (Claude Code), on behalf of @rswindell*
--- SBBSecho 3.37-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)