The Symptom

After a routine deployment, our Flink job started hanging. No errors, no timeouts—just stuck.

The Flink UI showed tasks in RUNNING state, but nothing was progressing. No failed checkpoints, no backpressure warnings. Just… frozen.

The Investigation

Step 1: Narrow Down the Scope

First, I checked which operator was stuck. The Flink DAG showed the job was blocked at a Doris sink—specifically during table creation. The DDL statement had been sent but never returned.

Was it a Flink problem or a Doris problem?

I ran a simple test: executed the same CREATE TABLE directly via Doris MySQL client.

1
2

CREATE TABLE test_table (...);
-- Hangs indefinitely

The DDL hung in the MySQL client too. Not a Flink issue—Doris FE was the problem.

Step 2: Check Doris FE

Logged into the Doris FE node. The FE process was alive, responding to simple queries like SHOW DATABASES. But any DDL operation hung forever.

Checked the FE logs—nothing unusual. No errors, no warnings about the stuck DDL.

Time to go deeper.

Step 3: strace the FE Process

When logs don’t help, strace is the next tool:

1

strace -f -p $(pgrep -f "DorisFeStart") 2>&1 | tee /tmp/fe-strace.log

Triggered another DDL and watched the output. Something caught my eye:

[pid 1053597] clone(...) = 1053598
[pid 1053598] execve("/usr/lib/jvm/java-17-openjdk-amd64/lib/jspawnhelper",
  ["/usr/lib/jvm/java-17-openjdk-amd64/lib/jspawnhelper",
   "17.0.16+8-Ubuntu-0ubuntu124.04.1",
   "946:947:949"],
  ...) = 0
[pid 1053598] +++ exited with 1 +++

A child process called jspawnhelper was spawned and immediately exited with code 1.

Step 4: A Suspicious Version Mismatch

I noticed the second argument: "17.0.16+8-Ubuntu-0ubuntu124.04.1". That looked like a JDK version string.

Checked the current system JDK:

1
2

java -version
# openjdk version "17.0.17+10-Ubuntu-124.04"

17.0.16 vs 17.0.17. The JVM was started with JDK 17.0.16, but the system had been upgraded to 17.0.17 since then.

I thought I found the root cause. But I was wrong.

The Plot Twist

After restarting Doris FE (which picked up the new JDK version), the jspawnhelper errors disappeared. But here’s the thing:

The two issues are unrelated.

To verify, I ran Doris FE with the mismatched jspawnhelper version for an extended period. The version check kept failing (confirmed via strace), but DDL operations worked fine. No hangs.

The original DDL hang had a different root cause—one we never identified. The jspawnhelper failure was a separate issue I stumbled upon during the investigation.

But the investigation wasn’t wasted. The jspawnhelper discovery led me down a fascinating rabbit hole about JDK internals that’s worth sharing.

What is jspawnhelper?

When Java calls Runtime.exec() or ProcessBuilder.start(), you might expect a simple fork() + exec(). But for large JVM processes, fork() is expensive:

• Even with copy-on-write, fork() must duplicate the entire page table
• A 32GB JVM heap means hundreds of megabytes just for page table duplication
• This makes process spawning slow and memory-intensive

JDK’s solution is jspawnhelper—a tiny helper binary:

JVM (32GB heap)
  └─ posix_spawn(jspawnhelper)     // Lightweight, fast
       └─ jspawnhelper (few MB)
            └─ exec(target_command) // Fork from small process

This pattern isn’t unique to Java. Chrome’s zygote, Android’s zygote, and Python’s forkserver all use the same idea: don’t fork from a bloated process—fork from a lightweight intermediary.

The Version Check Mechanism

The JVM and jspawnhelper communicate via a pipe, passing structured data like ChildStuff and SpawnInfo. If these structures change between versions, the communication breaks.

To prevent silent corruption, JDK added a strict version check:

1
2
3
4
5

// jspawnhelper.c
if (strcmp(argv[1], VERSION_STRING) != 0) {
    fprintf(stdout, "Incorrect Java version: %s\n", argv[1]);
    shutItDown();
}

The JVM passes its compiled-in version string as an argument:

1
2
3
4
5

// ProcessImpl_md.c
hlpargs[0] = (char*)helperpath;
hlpargs[1] = VERSION_STRING;  // Compiled at JDK build time
hlpargs[2] = buf1;            // File descriptors
hlpargs[3] = NULL;

When the system JDK gets upgraded, the jspawnhelper binary is replaced with a new version containing a different VERSION_STRING. But the running JVM still passes the old version → mismatch → exit(1).

The Backport Discovery

While investigating, I made an interesting discovery. I copied jspawnhelper from JDK 17.0.12 to the server, and suddenly shell commands worked fine.

Checked the 17.0.12 source—no version check logic existed. After some digging, I found JDK-8325621:

jspawnhelper is an internal JDK tool used by ProcessBuilder/Runtime.exec for process spawning. There’s an internal protocol between JDK and jspawnhelper that changes over time. Currently there’s no validation of version compatibility.

The version check was backported to JDK 17u after 17.0.12:

This is actually good design—fail fast is better than silent corruption. But it means minor JDK upgrades can now break running Java applications in ways they couldn’t before.

What We Learned

About jspawnhelper:

• JDK was upgraded from 17.0.16 to 17.0.17 while Doris FE was running
• jspawnhelper was failing with exit code 1 due to version mismatch
• Any Runtime.exec() call would fail silently (no subprocess output)
• The version check mechanism was backported in JDK-8325621

About the DDL hang:

• Root cause remains unknown
• Verified to be unrelated to jspawnhelper failure through extended testing
• Sometimes bugs disappear without explanation—and that’s frustrating but real

Takeaways

For Ops

1. Restart Java services after JDK upgrades—don’t assume minor versions are hot-swappable
2. Pin JDK versions—disable auto-updates or use containers
3. VMs are riskier than containers—container images lock versions; VMs get silent updates

For Developers

1. Always use timeouts with ProcessBuilder:

   1 2 3 4

   boolean finished = process.waitFor(30, TimeUnit.SECONDS); if (!finished) { process.destroyForcibly(); }

2. Handle subprocess failure gracefully—the child process might never produce output
3. Log when spawning external processes—it helps debugging when things go wrong

For Debugging

1. When Java apps behave strangely after system updates, think about jspawnhelper
2. strace -f is your friend—it shows what’s happening at the syscall level
3. Sometimes you find a real bug while chasing a different problem
4. Document what you don’t know—future you will thank you

The original DDL hang remains unsolved. But the detour into JDK internals was worth it. Sometimes debugging leads you somewhere unexpected—and that somewhere turns out to be more interesting than where you started.

References:

• JDK-8325621: Improve jspawnhelper version checks
• OpenJDK Commit