[BUG]: pybind11's initialization of internals needs to be made more thread safe in the presence of free threading

[EthanSteinberg]

Required prerequisites

• Make sure you've read the documentation. Your issue may be addressed there.
• Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• Consider asking first in the Gitter chat room or in a Discussion.

What version (or hash if on master) of pybind11 are you using?

a1d0091

Problem description

pybind11/include/pybind11/detail/internals.h

Line 498 in a1d0091

PYBIND11_NOINLINE internals &get_internals() {

is not thread safe under free-threading.

internals is a global singleton struct shared across all pybind11 modules. get_internals() either retrieves that current global singleton or creates a new one.

The problem is double initialization, where two copies of internals would be created by multiple modules being imported/initalized in different threads.

pybind11/include/pybind11/detail/internals.h

Lines 520 to 524 in a1d0091

	if (object internals_obj = get_internals_obj_from_state_dict(state_dict)) {
	internals_pp = get_internals_pp_from_capsule(internals_obj);
	}
	if (internals_pp && *internals_pp) {
	// We loaded the internals through `state_dict`, which means that our `error_already_set`

are the key problematic lines where we retrieve the current global, and initialize it if necessary.

The problem is that if two threads hit those lines at the same time, both threads could see that internals is null and both threads could initialize internals, which would lead to two copies.

Currently this is protected via the GIL (https://github.com/pybind/pybind11/blob/master/include/pybind11/detail/internals.h#L505), but that is problematic with the new python free threading setup.

In the short term this is still OK because the GIL is re-enabled during module import, but that is not a good long term strategy.

Reproducible example code

No response

Is this a regression? Put the last known working version here if it is.

Not a regression

[EthanSteinberg]

PyDict_SetDefaultRef appears to be the key for making this actually safe. It enables a conditional set on a dictionary.

[rwgk]

Extra background (to the best of my knowledge, please correct if necessary):

• fix: support free-threaded CPython with GIL disabled #5148 added mutexes around internals access, but did not add any for internals creation.

• get_internals() is called from here (PYBIND11_MODULE() macro) during module import (which is still protected by the GIL, even with free-threading enabled):

  pybind11/include/pybind11/detail/common.h

  Line 489 in fc59f4e

  PYBIND11_ENSURE_INTERNALS_READY \

I believe it would be good to work this into the PR description, and to explain the situations in which get_internals() might be called without being protected either by a mutex or the module import protections.