test_codecs: LocaleCodecTest fails with some locales on FreeBSD if UTF-8 Mode is disabled

Example on FreeBSD 16 with `be_BY.CP1251` locale which uses the `cp1251` encoding.

I can reproduce the failure on FreeBSD 15 and FreeBSD 16.

Logs:

<details>

```
vstinner@freebsd16$ LC_ALL=be_BY.CP1251 ./python -X utf8=0 -m test -v test_codecs -m LocaleCodecTest 
== CPython 3.16.0a0 (heads/main:85c03d47a24, Oct 6 2026, 04:22:51) [Clang 21.1.8 (https://github.com/llvm/llvm-project.git llvmorg-21.1.8-0-g2078da43e25a)]
== FreeBSD-16.0-CURRENT-amd64-64bit-ELF little-endian
== Python build: debug
== cwd: /home/vstinner/python/main/build/test_python_worker_15409�
== CPU count: 6
== encodings: locale=CP1251 FS=cp1251
== resources: all test resources are disabled, use -u option to unskip tests

Using random seed: 2716217037
0:00:00 load avg: 0.27 mem: 29.8 MiB Run 1 test sequentially in a single process
0:00:00 load avg: 0.27 mem: 29.8 MiB [1/1] test_codecs
test_decode_strict (test.test_codecs.LocaleCodecTest.test_decode_strict) ... 
  test_decode_strict (test.test_codecs.LocaleCodecTest.test_decode_strict) (encoded=b'blatin1:\xa7\xe9') ... FAIL
  test_decode_strict (test.test_codecs.LocaleCodecTest.test_decode_strict) (encoded=b'b255:\xff') ... FAIL
test_decode_surrogateescape (test.test_codecs.LocaleCodecTest.test_decode_surrogateescape) ... 
  test_decode_surrogateescape (test.test_codecs.LocaleCodecTest.test_decode_surrogateescape) (encoded=b'blatin1:\xa7\xe9') ... FAIL
  test_decode_surrogateescape (test.test_codecs.LocaleCodecTest.test_decode_surrogateescape) (encoded=b'b255:\xff') ... FAIL
test_decode_surrogatepass (test.test_codecs.LocaleCodecTest.test_decode_surrogatepass) ... skipped "'cp1251' decoder doesn't support surrogatepass error handler"
test_decode_unsupported_error_handler (test.test_codecs.LocaleCodecTest.test_decode_unsupported_error_handler) ... ok
test_encode_strict (test.test_codecs.LocaleCodecTest.test_encode_strict) ... 
  test_encode_strict (test.test_codecs.LocaleCodecTest.test_encode_strict) (text='ulatin1:�\xe9') ... FAIL
  test_encode_strict (test.test_codecs.LocaleCodecTest.test_encode_strict) (text='u255:\xff') ... FAIL
  test_encode_strict (test.test_codecs.LocaleCodecTest.test_encode_strict) (text='UCS:\xe9�\U0010ffff') ... FAIL
test_encode_surrogateescape (test.test_codecs.LocaleCodecTest.test_encode_surrogateescape) ... 
  test_encode_surrogateescape (test.test_codecs.LocaleCodecTest.test_encode_surrogateescape) (text='ulatin1:�\xe9') ... FAIL
  test_encode_surrogateescape (test.test_codecs.LocaleCodecTest.test_encode_surrogateescape) (text='u255:\xff') ... FAIL
  test_encode_surrogateescape (test.test_codecs.LocaleCodecTest.test_encode_surrogateescape) (text='UCS:\xe9�\U0010ffff') ... FAIL
test_encode_surrogatepass (test.test_codecs.LocaleCodecTest.test_encode_surrogatepass) ... skipped "'cp1251' encoder doesn't support surrogatepass error handler"
test_encode_unsupported_error_handler (test.test_codecs.LocaleCodecTest.test_encode_unsupported_error_handler) ... ok
test_memory_error (test.test_codecs.LocaleCodecTest.test_memory_error) ... ok

======================================================================
FAIL: test_decode_strict (test.test_codecs.LocaleCodecTest.test_decode_strict) (encoded=b'blatin1:\xa7\xe9')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/python/main/Lib/test/test_codecs.py", line 4208, in check_decode_strings
    self.assertEqual(decoded, expected)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
AssertionError: 'blatin1:�\xe9' != 'blatin1:��'
- blatin1:�\xe9
?          ^
+ blatin1:��
?          ^


======================================================================
FAIL: test_decode_strict (test.test_codecs.LocaleCodecTest.test_decode_strict) (encoded=b'b255:\xff')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/python/main/Lib/test/test_codecs.py", line 4208, in check_decode_strings
    self.assertEqual(decoded, expected)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
AssertionError: 'b255:\xff' != 'b255:�'
- b255:\xff
?      ^
+ b255:�
?      ^


======================================================================
FAIL: test_decode_surrogateescape (test.test_codecs.LocaleCodecTest.test_decode_surrogateescape) (encoded=b'blatin1:\xa7\xe9')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/python/main/Lib/test/test_codecs.py", line 4208, in check_decode_strings
    self.assertEqual(decoded, expected)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
AssertionError: 'blatin1:�\xe9' != 'blatin1:��'
- blatin1:�\xe9
?          ^
+ blatin1:��
?          ^


======================================================================
FAIL: test_decode_surrogateescape (test.test_codecs.LocaleCodecTest.test_decode_surrogateescape) (encoded=b'b255:\xff')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/python/main/Lib/test/test_codecs.py", line 4208, in check_decode_strings
    self.assertEqual(decoded, expected)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
AssertionError: 'b255:\xff' != 'b255:�'
- b255:\xff
?      ^
+ b255:�
?      ^


======================================================================
FAIL: test_encode_strict (test.test_codecs.LocaleCodecTest.test_encode_strict) (text='ulatin1:�\xe9')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/python/main/Lib/test/test_codecs.py", line 4091, in check_encode_strings
    expected = text.encode(self.ENCODING, errors)
  File "/home/vstinner/python/main/Lib/encodings/cp1251.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_table)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode character '\xe9' in position 9: character maps to <undefined>
encoding with 'cp1251' codec failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/vstinner/python/main/Lib/test/test_codecs.py", line 4111, in check_encode_strings
    with self.assertRaises(RuntimeError) as cm:
         ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
AssertionError: RuntimeError not raised

======================================================================
FAIL: test_encode_strict (test.test_codecs.LocaleCodecTest.test_encode_strict) (text='u255:\xff')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/python/main/Lib/test/test_codecs.py", line 4091, in check_encode_strings
    expected = text.encode(self.ENCODING, errors)
  File "/home/vstinner/python/main/Lib/encodings/cp1251.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_table)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode character '\xff' in position 5: character maps to <undefined>
encoding with 'cp1251' codec failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/vstinner/python/main/Lib/test/test_codecs.py", line 4111, in check_encode_strings
    with self.assertRaises(RuntimeError) as cm:
         ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
AssertionError: RuntimeError not raised

======================================================================
FAIL: test_encode_strict (test.test_codecs.LocaleCodecTest.test_encode_strict) (text='UCS:\xe9�\U0010ffff')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/python/main/Lib/test/test_codecs.py", line 4091, in check_encode_strings
    expected = text.encode(self.ENCODING, errors)
  File "/home/vstinner/python/main/Lib/encodings/cp1251.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_table)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode character '\xe9' in position 4: character maps to <undefined>
encoding with 'cp1251' codec failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/vstinner/python/main/Lib/test/test_codecs.py", line 4114, in check_encode_strings
    self.assertEqual(str(cm.exception), errmsg)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'encode error: pos=5' != 'encode error: pos=4'
- encode error: pos=5
?                   ^
+ encode error: pos=4
?                   ^


======================================================================
FAIL: test_encode_surrogateescape (test.test_codecs.LocaleCodecTest.test_encode_surrogateescape) (text='ulatin1:�\xe9')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/python/main/Lib/test/test_codecs.py", line 4091, in check_encode_strings
    expected = text.encode(self.ENCODING, errors)
  File "/home/vstinner/python/main/Lib/encodings/cp1251.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_table)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode character '\xe9' in position 9: character maps to <undefined>
encoding with 'cp1251' codec failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/vstinner/python/main/Lib/test/test_codecs.py", line 4106, in check_encode_strings
    with self.assertRaises(RuntimeError) as cm:
         ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
AssertionError: RuntimeError not raised

======================================================================
FAIL: test_encode_surrogateescape (test.test_codecs.LocaleCodecTest.test_encode_surrogateescape) (text='u255:\xff')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/python/main/Lib/test/test_codecs.py", line 4091, in check_encode_strings
    expected = text.encode(self.ENCODING, errors)
  File "/home/vstinner/python/main/Lib/encodings/cp1251.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_table)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode character '\xff' in position 5: character maps to <undefined>
encoding with 'cp1251' codec failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/vstinner/python/main/Lib/test/test_codecs.py", line 4106, in check_encode_strings
    with self.assertRaises(RuntimeError) as cm:
         ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
AssertionError: RuntimeError not raised

======================================================================
FAIL: test_encode_surrogateescape (test.test_codecs.LocaleCodecTest.test_encode_surrogateescape) (text='UCS:\xe9�\U0010ffff')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/python/main/Lib/test/test_codecs.py", line 4091, in check_encode_strings
    expected = text.encode(self.ENCODING, errors)
  File "/home/vstinner/python/main/Lib/encodings/cp1251.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_table)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode character '\xe9' in position 4: character maps to <undefined>
encoding with 'cp1251' codec failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/vstinner/python/main/Lib/test/test_codecs.py", line 4109, in check_encode_strings
    self.assertEqual(str(cm.exception), errmsg)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'encode error: pos=5' != 'encode error: pos=4'
- encode error: pos=5
?                   ^
+ encode error: pos=4
?                   ^


----------------------------------------------------------------------
Ran 9 tests in 0.052s

FAILED (failures=10, skipped=2)
test test_codecs failed
0:00:00 load avg: 0.27 mem: 44.5 MiB [1/1/1] test_codecs failed (10 failures)

== Tests result: FAILURE ==

1 test failed:
    test_codecs

Total duration: 369 ms
Total tests: run=9 (filtered) failures=10 skipped=2
Total test files: run=1/1 (filtered) failed=1
Result: FAILURE
```

</details>

The problem is that the Python codec (of `locale.getencoding()` encoding) cannot encode some characters, whereas Py_EncodeLocale() can:

```
vstinner@freebsd16$ LC_ALL=be_BY.CP1251 ./python -X utf8=0
Python 3.16.0a0 (heads/main:85c03d47a24, Oct  6 2026, 04:22:51) [Clang 21.1.8 (https://github.com/llvm/llvm-project.git llvmorg-21.1.8-0-g2078da43e25a)] on freebsd

>>> import _testlimitedcapi, _testinternalcapi, locale
>>> enc=locale.getencoding(); enc
'CP1251'

>>> s = '\xe9'
>>> s.encode(enc)
UnicodeEncodeError: 'charmap' codec can't encode character '\xe9' in position 0: character maps to <undefined>
encoding with 'CP1251' codec failed

>>> _testlimitedcapi.encode_locale(s) # Py_EncodeLocale()
b'\xe9'
>>> _testinternalcapi.encode_locale(s, 0, 'strict') # _Py_EncodeLocale()
b'\xe9'
```

Navigation Menu

Repository navigation

test_codecs: LocaleCodecTest fails with some locales on FreeBSD if UTF-8 Mode is disabled #158890

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

test_codecs: LocaleCodecTest fails with some locales on FreeBSD if UTF-8 Mode is disabled #158890

Description

Activity

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions