Skip to content

stream: fix Utf8Stream flush handling - #66473

Open
jasnell wants to merge 1 commit into
nodejs:mainfrom
jasnell:jasnell/utf8stream-flush-fix
Open

jasnell wants to merge 1 commit into
nodejs:mainfrom
jasnell:jasnell/utf8stream-flush-fix

Conversation

@jasnell

@jasnell jasnell commented Oct 3, 2026

Copy link
Copy Markdown
Member

Fix several issues with Utf8Stream flushing:

  • flush() now writes buffered data regardless of minLength and invokes the callback only after pending writes complete, including when minLength is zero and a write is in flight.
  • Multiple concurrent flush() calls are tracked correctly and end() waits for pending flushes before closing.
  • flushSync() throws ERR_INVALID_STATE if called while an asynchronous write is in progress instead of corrupting output.
  • Periodic flushes no longer stack up while a flush is pending.
  • fsync is skipped for stdout/stderr file descriptors.
  • reopen(), end() and destroy() behave correctly when the stream is destroyed while still opening.

Separated out from #65840

@jasnell
jasnell requested review from mcollina and ronag October 3, 2026 01:48
@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

Review requested:

  • @nodejs/streams

@nodejs-github-bot nodejs-github-bot added needs-ci PRs that need a full CI run. stream Issues and PRs related to Node.js streams. labels Oct 3, 2026
@codecov

codecov Bot commented Oct 3, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.40741% with 34 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.42%. Comparing base (9746ebc) to head (6a11fbc).
⚠️ Report is 14 commits behind head on main.

Files with missing lines Patch % Lines
lib/internal/streams/fast-utf8-stream.js 87.40% 34 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #66473      +/-   ##
==========================================
+ Coverage   90.40%   90.42%   +0.02%     
==========================================
  Files         791      791              
  Lines      276120   276315     +195     
  Branches    53022    53085      +63     
==========================================
+ Hits       249618   249853     +235     
+ Misses      16897    16871      -26     
+ Partials     9605     9591      -14     
Files with missing lines Coverage Δ
lib/internal/streams/fast-utf8-stream.js 88.62% <87.40%> (+5.55%) ⬆️

... and 31 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@mcollina

This comment was marked as resolved.

@mcollina

This comment was marked as outdated.

@jasnell
jasnell force-pushed the jasnell/utf8stream-flush-fix branch from 8034879 to d52d366 Compare October 4, 2026 11:13
@jasnell

jasnell commented Oct 4, 2026

Copy link
Copy Markdown
Member Author

... why? If it’s a file, this is useful.

You're right. changing.

Updated for both issues

mcollina added a commit to pinojs/sonic-boom that referenced this pull request Oct 4, 2026
- flush() with minLength 0 now waits for in-flight writes before
  calling the callback.
- end() waits for pending flush callbacks before closing.
- Concurrent flush() calls are tracked with counters.
- flush() no longer fsyncs stdout/stderr and ignores EBADF.
- Periodic flushes do not stack while a flush is pending.
- end()/reopen()/destroy() while opening no longer double-close or throw;
  pending flushes fail with an error when the stream is destroyed.

The flushSync() change from the upstream PR (throwing while a write
is in progress) is intentionally left out as it is breaking.
mcollina added a commit to pinojs/sonic-boom that referenced this pull request Oct 4, 2026
- flushSync() preserves ordering where it can: it cancels a pending
  EAGAIN/EBUSY retry and writes everything in order, writes the
  remainder of a partial write first when called from a 'write'
  listener, and writes to the previous file while reopening. With an
  async write in flight, only the queued data is written.
- 'write' is emitted after the written bytes are released.
- flushBufferSync() keeps _lens in sync when re-queuing _writingBuf.
- flush() fsyncs stdout/stderr again (useful when redirected to a
  file) and ignores EBADF/EINVAL/ENOTSUP/EOPNOTSUPP/EROFS from fsync.

@mcollina mcollina left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

mcollina added a commit to pinojs/sonic-boom that referenced this pull request Oct 5, 2026
Align with nodejs/node#66473: when destroy() is called while a write
is in flight, a pending flush() now succeeds if all the data was
written, and fails only if data is left. A pending EAGAIN/EBUSY retry
is cancelled on destroy instead of writing to the closed fd.
@mcollina

mcollina commented Oct 5, 2026

Copy link
Copy Markdown
Member

This needs a rebase now

mcollina added a commit to pinojs/sonic-boom that referenced this pull request Oct 5, 2026
* fix: flush handling (port of nodejs/node#66473)

- flush() with minLength 0 now waits for in-flight writes before
  calling the callback.
- end() waits for pending flush callbacks before closing.
- Concurrent flush() calls are tracked with counters.
- flush() no longer fsyncs stdout/stderr and ignores EBADF.
- Periodic flushes do not stack while a flush is pending.
- end()/reopen()/destroy() while opening no longer double-close or throw;
  pending flushes fail with an error when the stream is destroyed.

The flushSync() change from the upstream PR (throwing while a write
is in progress) is intentionally left out as it is breaking.

* fix: port updated flushSync and fsync handling from nodejs/node#66473

- flushSync() preserves ordering where it can: it cancels a pending
  EAGAIN/EBUSY retry and writes everything in order, writes the
  remainder of a partial write first when called from a 'write'
  listener, and writes to the previous file while reopening. With an
  async write in flight, only the queued data is written.
- 'write' is emitted after the written bytes are released.
- flushBufferSync() keeps _lens in sync when re-queuing _writingBuf.
- flush() fsyncs stdout/stderr again (useful when redirected to a
  file) and ignores EBADF/EINVAL/ENOTSUP/EOPNOTSUPP/EROFS from fsync.

* fix: settle pending flushes after an in-flight write on destroy

Align with nodejs/node#66473: when destroy() is called while a write
is in flight, a pending flush() now succeeds if all the data was
written, and fails only if data is left. A pending EAGAIN/EBUSY retry
is cancelled on destroy instead of writing to the closed fd.
mcollina added a commit to pinojs/sonic-boom that referenced this pull request Oct 5, 2026
Align with nodejs/node#66473: destroy() while the file is opening now
sets `destroyed` right away, so write()/end()/flush() behave as on any
destroyed stream instead of silently accepting data that is dropped.
A pending flush() is failed with an 'error' event.
Fix several issues with `Utf8Stream` flushing:

* `flush()` now writes buffered data regardless of `minLength`
  and invokes the callback only after pending writes complete,
  including when `minLength` is zero and a write is in flight.
* Multiple concurrent `flush()` calls are tracked correctly and
  `end()` waits for pending flushes before closing.
* `flushSync()` throws `ERR_INVALID_STATE` if called while an
  asynchronous write is in progress instead of corrupting output.
* Periodic flushes no longer stack up while a flush is pending.
* `fsync` is skipped for stdout/stderr file descriptors.
* `reopen()`, `end()` and `destroy()` behave correctly when the
  stream is destroyed while still opening.

Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode
@jasnell
jasnell force-pushed the jasnell/utf8stream-flush-fix branch from d52d366 to 6a11fbc Compare October 5, 2026 21:59
@jasnell

jasnell commented Oct 5, 2026

Copy link
Copy Markdown
Member Author

@mcollina rebased... PTAL

@mcollina mcollina left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@mcollina mcollina added the request-ci Add this label to start a Jenkins CI on a PR. Only starts once the PR has an approving review. label Oct 6, 2026
@github-actions github-actions Bot removed the request-ci Add this label to start a Jenkins CI on a PR. Only starts once the PR has an approving review. label Oct 6, 2026
@nodejs-github-bot

This comment was marked as outdated.

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

@mcollina mcollina added the resume-ci Add this label to resume the latest eligible Jenkins CI run on a PR with an approving review. label Oct 7, 2026
@github-actions github-actions Bot removed the resume-ci Add this label to resume the latest eligible Jenkins CI run on a PR with an approving review. label Oct 7, 2026
@github-actions github-actions Bot added the resume-ci-failed Resuming CI with the resume-ci label failed and requires manual intervention. label Oct 7, 2026
@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

Failed to resume CI

   ✖  test/parallel/test-fastutf8stream-flush-sync.js
   ℹ  https://ci.nodejs.org/job/node-test-commit-osx/nodes=macos15-x64/73973/consoleText
✖  Refusing to resume CI: failures reference files changed by this PR
Full Auto Start CI output
�[36m⠋�[39m Validating Jenkins credentials
�[36m⠋�[39m Validating Jenkins credentials
✔  Jenkins credentials valid
�[36m⠙�[39m Looking for CI runs for pull request 66473
�[36m⠙�[39m Looking for CI runs for pull request 66473
�[36m⠙�[39m Getting PR from nodejs/node/pull/66473
�[36m⠙�[39m Getting reviews from nodejs/node/pull/66473
�[36m⠙�[39m Getting comments from nodejs/node/pull/66473
✔  Found PR CI job 78233
�[36m⠹�[39m Querying data for job/node-test-pull-request/78233/
�[36m⠹�[39m Querying data for job/node-test-pull-request/78233/
�[36m⠹�[39m Querying API for job/node-test-pull-request/78233/
✔  Build data downloaded
�[36m⠹�[39m Checking whether PR CI job 78233 can be resumed
�[36m⠹�[39m Checking whether PR CI job 78233 can be resumed
✔  Jenkins offers a Resume build action
�[36m⠸�[39m Checking failures against changed PR files
�[36m⠸�[39m Checking failures against changed PR files
�[36m⠼�[39m Querying data for job/node-test-pull-request/78233/
�[36m⠼�[39m Querying API for job/node-test-pull-request/78233/
✔  Build data downloaded
�[36m⠼�[39m Querying failures of job/node-test-commit/93073/
�[36m⠼�[39m Querying failures of job/node-test-commit/93073/
�[36m⠼�[39m Querying API for job/node-test-commit-osx/73973/
�[36m⠴�[39m Querying API for job/node-test-commit-osx/nodes=macos15-x64/73973/
�[36m⠴�[39m Querying console text for job/node-test-commit-osx/nodes=macos15-x64/73973/
✔  Data downloaded
   ✖  test/parallel/test-fastutf8stream-flush-sync.js
   ℹ  https://ci.nodejs.org/job/node-test-commit-osx/nodes=macos15-x64/73973/consoleText
not ok 1199 parallel/test-fastutf8stream-flush-sync
  ---
  duration_ms: 5287.55500
  severity: fail
  exitcode: 1
  stack: |-
    Mismatched <anonymous> function calls. Expected exactly 1, actual 2.
        at Proxy.mustCall (/Users/admin/build/workspace/node-test-commit-osx/nodes/macos15-x64/test/common/index.js:470:10)
        at Object.<anonymous> (/Users/admin/build/workspace/node-test-commit-osx/nodes/macos15-x64/test/parallel/test-fastutf8stream-flush-sync.js:135:19)
        at Module._compile (node:internal/modules/cjs/loader:1968:14)
        at Object..js (node:internal/modules/cjs/loader:2108:10)
        at Module.load (node:internal/modules/cjs/loader:1690:32)
        at Module._load (node:internal/modules/cjs/loader:1480:12)
        at wrapModuleLoad (node:internal/modules/cjs/loader:261:19)
        at Module.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:173:5)
        at node:internal/main/run_main_module:33:47
    Mismatched <anonymous> function calls. Expected exactly 1, actual 2.
        at Proxy.mustCall (/Users/admin/build/workspace/node-test-commit-osx/nodes/macos15-x64/test/common/index.js:470:10)
        at Object.<anonymous> (/Users/admin/build/workspace/node-test-commit-osx/nodes/macos15-x64/test/parallel/test-fastutf8stream-flush-sync.js:135:19)
        at Module._compile (node:internal/modules/cjs/loader:1968:14)
        at Object..js (node:internal/modules/cjs/loader:2108:10)
        at Module.load (node:internal/modules/cjs/loader:1690:32)
        at Module._load (node:internal/modules/cjs/loader:1480:12)
        at wrapModuleLoad (node:internal/modules/cjs/loader:261:19)
        at Module.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:173:5)
        at node:internal/main/run_main_module:33:47
  ...
✖  Refusing to resume CI: failures reference files changed by this PR

View workflow run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-ci PRs that need a full CI run. resume-ci-failed Resuming CI with the resume-ci label failed and requires manual intervention. stream Issues and PRs related to Node.js streams.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants