Skip to content

Do not resend a consumed multipart stream part - #2398

Merged
hyperxpro merged 2 commits into
AsyncHttpClient:mainfrom
mkurz:fix/multipart-stream-replay
Nov 22, 2026
Merged

hyperxpro merged 2 commits into
AsyncHttpClient:mainfrom
mkurz:fix/multipart-stream-replay

Conversation

@mkurz

@mkurz mkurz commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • A retry, a redirect that keeps the body, or a replay by a ResponseFilter or IOExceptionFilter writes a new multipart body over the same InputStreamPart streams, and reads them from wherever the earlier attempt left them. An InputStream body already fails cleanly in that case (Failed stream request hangs until timeout #1973); multipart had no such check.
  • With the default configuration, AHC retries a request whose connection the server closed without a response. With an InputStreamPart:
    • No length, and a stream that returns end of stream once read (a ByteArrayInputStream, or a wrapper that ignores close()): the retry sends an empty part, the server answers, and the request succeeds.
    • A known length: the retry declares the full Content-Length but sends only the part headers, so it waits for a timeout.
    • A FileInputStream: the retry fails with a ClosedChannelException.
  • The redirect check from fix(netty): preserve all redirect body types #2316 rejects every InputStreamPart, even after a redirect before 100 Continue, when nothing was sent yet.

Change

  • NettyResponseFuture: records, by identity, the stream of each part that a write of the request has started on (markStreamPartConsumed, isStreamPartConsumed). The set is only created for a request with stream parts.
  • NettyRequestSender:
    • Before the HTTP/1.1 and HTTP/2 writes of a multipart body, it records the streams of the request's InputStreamParts.
    • sendNextRequest, which retries, redirects, authentication retries and replays go through, fails the request with IOException: Multipart InputStream body part 'file' already consumed and cannot be sent again if one of its InputStreamParts has a recorded stream. It does so before connecting, so nothing of that request is sent.
      • It only checks the parts that the request sends. A CONNECT sends none, and neither does a request with a body of higher precedence, which setBodyParts leaves in place, such as a string body that a filter set for its replay.
      • A connection kept for the request, as after a 401 or 407 on a keep-alive connection, is closed, so it does not keep its slot under maxConnections.
    • The streams of parts that a filter supplies for a replay were not recorded, so they are sent as they are.
  • Redirect30xInterceptor: rejects an InputStreamPart only if a write has started on its stream. A redirect before 100 Continue keeps the part again.
  • A consumed stream is not reset, even if it supports mark/reset: the earlier write may still be reading it, the part closes its stream once written, and the stream's mark need not be where the earlier write started.
  • The flag that tracks an InputStream body (isStreamConsumed) is unchanged, and multipart does not use it.
  • Tests:
    • MultipartStreamPartReplayTest:
      • the server reads the whole request and closes without a response: the retry fails, also for a stream that supports mark/reset and for one whose mark is before its start, and the server receives no second request;
      • a 401 on a keep-alive connection: the authentication retry fails, and with maxConnections at 1 the next request is sent;
      • a ResponseFilter replaces the parts for its replay: a fresh stream part is sent, from its current position even if it supports mark/reset; a reused stream part next to a fresh one fails the replay, and the server receives no second request;
      • a ResponseFilter sets a string body for its replay and keeps the consumed parts: the string body is sent; it changes the method to CONNECT and keeps the parts: the CONNECT is sent.
    • Http2MultipartStreamPartReplayTest: a ResponseFilter replay over HTTP/2 fails for a consumed stream part, and the server receives no second HEADERS frame; a fresh stream part from the filter is sent.
    • RedirectBodyTest:
      • a stream part kept by a 307 before 100 Continue, without being read before the redirect;
      • a 307 that arrives while the stream part is being uploaded fails the redirect: the server stops reading after the first KiB of a 32 MiB part, and the client's send buffer and the server's receive buffer are set to 64 KiB, so the upload cannot complete before the 307 arrives; the test checks that the part's stream was not fully read;
      • the existing test for a stream part whose body was sent before the 307 also runs with a BufferedInputStream over the file.
    • NettyRequestSenderConsumedBodyRetryTest: a ResponseFilter replay with a consumed stream part fails promptly.

Notes

  • As for an InputStream body, a stream counts as consumed once the body write starts, not once the stream is read. A retry after a write that failed before it reached the stream part therefore fails too.
  • Unlike an InputStream body (Failed stream request hangs until timeout #1973), a consumed stream part is not reset even if it supports mark/reset. A ResponseFilter or IOExceptionFilter can replay such a request with new parts.
  • NettyResponseFuture.markStreamPartConsumed and isStreamPartConsumed are new public methods.

AI disclosure

Claude Code on behalf of Matthias Kurz. The commits include Co-Authored-By per AGENTS.md.

Test plan

  • ./mvnw -B -ntp -Dgpg.skip=true clean verify on JDK 11, on this branch and on this branch merged with main at 3d0a347: 2,222 and 2,228 tests, 0 failures, 0 errors, 22 skipped; Revapi passed. Docker was not running, so the 22 skipped include the Testcontainers proxy tests. No test-skipping flags were used.
  • The classes above, together with NettyInputStreamBodyTest and RedirectRefusalTest: 113 tests, 0 failures, on JDK 11, 17, 21 and 25.
  • Without the change in client/src/main, the retry tests, the authentication retry test, the tests that reuse a stream part over HTTP/1.1 and HTTP/2, the replay test in NettyRequestSenderConsumedBodyRetryTest and the redirect test before 100 Continue fail: the retries and replays succeed or time out, and the redirect fails with cannot be replayed after redirect. The tests with only fresh stream parts, a string body or a CONNECT pass on main as well, and so do the redirect tests that expect a refusal, as main refuses every stream part on redirect.
  • On 4488f13, the previous commit of this PR, the 307 during the upload fails with a ClosedChannelException instead of the refusal, the BufferedInputStream redirect fails after connecting to the target with already consumed and cannot be reset for a retry, and the retries, authentication retries and replays of a consumed stream that supports mark/reset succeed, including the one whose mark is before its start.
  • The stream part redirect tests, MultipartStreamPartReplayTest and Http2MultipartStreamPartReplayTest: 10 runs, 0 failures.

A retry, a redirect that keeps the body, or a replay by a filter
builds a new multipart body over the same InputStreamPart streams and
reads them from wherever the first attempt left them. With the
default configuration, AHC retries a request whose connection the
server closed without a response. If the part had no length and its
stream returns end of stream once read, as a ByteArrayInputStream
does, the retry sent an empty part and the server's answer completed
the request. With a known length, the retry declared the full length
but sent only the part headers, so it waited for a timeout. With a
FileInputStream, it failed with a ClosedChannelException.

NettyResponseFuture now records, by identity, the stream of each part
that a write of the request has started on. A later write that finds
a recorded stream resets it if it supports mark/reset, as
NettyInputStreamBody does for an InputStream body, and otherwise
fails the request with an IOException before the body is written.
The streams of parts that a filter supplies for a replay are sent as
they are. NettyBodyBody.write does not declare IOException, so
NettyRequestSender runs the check before the HTTP/1.1 and HTTP/2
writes.

The redirect check now rejects an InputStreamPart only if a write has
started on its stream and the stream does not support mark/reset. A
redirect before 100 Continue, when nothing was sent, keeps the part
again, as it did before AsyncHttpClient#2316, and a stream that supports mark/reset
is replayed.

Claude Code on behalf of Matthias Kurz

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A retry, redirect or replay reset the stream of an InputStreamPart
that an earlier write of the request had started on, if the stream
supported mark/reset. That is not safe:

- The earlier write may still be reading the stream, when a 307
  arrives during the upload.
- The part closes its stream once written, so a BufferedInputStream
  over a file passed the redirect check and then failed on the new
  connection.
- The mark may be before where the earlier write started, and the
  part then writes more than its declared length.

Such a stream is no longer reset. NettyRequestSender.sendNextRequest
fails the request before connecting when one of the InputStreamParts
it sends was consumed, so nothing of it is sent, over HTTP/1.1 or
HTTP/2. Parts that it does not send, for a CONNECT or behind a body
with higher precedence, are not checked. A connection kept for the
request, as after a 401 on a keep-alive connection, is closed. The
redirect check rejects a consumed stream part again, whatever its
mark/reset support, and still keeps a part whose write had not
started. The streams of parts that a filter supplies for a replay are
still sent.

Claude Code on behalf of Matthias Kurz

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mkurz
mkurz requested a review from hyperxpro November 21, 2026 22:10
@hyperxpro
hyperxpro merged commit f927b80 into AsyncHttpClient:main Nov 22, 2026
13 checks passed
@mkurz
mkurz deleted the fix/multipart-stream-replay branch November 22, 2026 14:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants