Skip to content

Stash pybind11 data structures in interpreter state dictionary #4293

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wjakob wants to merge 10 commits into from
Closed

Stash pybind11 data structures in interpreter state dictionary #4293

Changes from 2 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
3b14721
stash pybind11 data structures in interpreter state dictionary
wjakob Oct 28, 2022
7a94406
style: pre-commit fixes
pre-commit-ci[bot] Oct 28, 2022
3a89f67
Make changes specific to Python 3.8+, incorporate feedback
wjakob Oct 31, 2022
daf768a
fix segfault
wjakob Oct 31, 2022
8df2e81
fix C++ test
wjakob Nov 2, 2022
f4e6c78
style: pre-commit fixes
pre-commit-ci[bot] Nov 2, 2022
cc03a56
continue to use PyEval_GetBuiltins for PyPy
wjakob Nov 2, 2022
780c6ff
incorporated feedback
wjakob Nov 2, 2022
39c21a3
style: pre-commit fixes
pre-commit-ci[bot] Nov 2, 2022
78e074b
clang-tidy fix
wjakob Nov 2, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 26 additions & 9 deletions include/pybind11/detail/internals.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -277,6 +277,12 @@ struct type_info {
# endif
#endif

#if PY_VERSION_HEX < 0x03090000
# define PYBIND11_INTERPRETER_STATE_GET _PyInterpreterState_Get
#else
# define PYBIND11_INTERPRETER_STATE_GET PyInterpreterState_Get
#endif

#define PYBIND11_INTERNALS_ID \
"__pybind11_internals_v" PYBIND11_TOSTRING(PYBIND11_INTERNALS_VERSION) \
PYBIND11_INTERNALS_KIND PYBIND11_COMPILER_TYPE PYBIND11_STDLIB PYBIND11_BUILD_ABI \
Expand Down Expand Up @@ -403,7 +409,7 @@ inline void translate_local_exception(std::exception_ptr p) {

/// Return a reference to the current `internals` data
PYBIND11_NOINLINE internals &get_internals() {
auto **&internals_pp = get_internals_pp();
internals **&internals_pp = get_internals_pp();
if (internals_pp && *internals_pp) {
return **internals_pp;
}
Expand All @@ -419,11 +425,24 @@ PYBIND11_NOINLINE internals &get_internals() {
} gil;
error_scope err_scope;

PYBIND11_STR_TYPE id(PYBIND11_INTERNALS_ID);
auto builtins = handle(PyEval_GetBuiltins());
if (builtins.contains(id) && isinstance<capsule>(builtins[id])) {
internals_pp = static_cast<internals **>(capsule(builtins[id]));
const char *id_cstr = PYBIND11_INTERNALS_ID;
PYBIND11_STR_TYPE id(id_cstr);

dict state_dict
= reinterpret_borrow<dict>(PyInterpreterState_GetDict(PYBIND11_INTERPRETER_STATE_GET()));
if (!state_dict)
pybind11_fail("get_internals(): PyInterpreterState_GetDict() failed!");

if (state_dict.contains(id_cstr)) {
object o = state_dict[id];
// May fail if 'capsule_obj' is not a capsule, or if it has a different
// name. We clear the error status below in that case
internals_pp = static_cast<internals **>(PyCapsule_GetPointer(o.ptr(), id_cstr));
Copy link
Collaborator

@Skylion007 Skylion007 Nov 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why the raw CAPI here? Any particular reason we are changing it from pytype.h API it was before?

Copy link
Collaborator

@rwgk rwgk Nov 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is my current version:

        void *raw_ptr = PyCapsule_GetPointer(state_dict[id].ptr(), id_cstr);
        if (raw_ptr == nullptr) {
            raise_from(
                PyExc_SystemError,
                "pybind11::detail::get_internals(): Retrieve internals** from capsule FAILED");
        }
        internals_pp = static_cast<internals **>(raw_ptr);

The nice thing is that PyCapsule_GetPointer() does everything "just right" in one simple line. Additionally, in this particular situation I'd definitely want to avoid the throw PYBIND11_OBJECT_CHECK_FAILED() that comes with the capsule constructor.

I don't have a good idea for using capsule in an elegant way here TBH.

Copy link
Collaborator

@rwgk rwgk Nov 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While looking around more (work under PR #4329) I noticed this code in embed.h (finalize_interpreter()):

    handle builtins(PyEval_GetBuiltins());
    ...
    if (builtins.contains(id) && isinstance<capsule>(builtins[id])) {
        internals_ptr_ptr = capsule(builtins[id]);
    }

Two things learned:

  • That needs to be changed, too, to inspect the correct dict.
  • The implementation is a bit on the high-level overkill side, scraping by an if (...) throw, but it is elegant!

if (!internals_pp)
PyErr_Clear();
Copy link
Collaborator

@rwgk rwgk Nov 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR (with the #4307 tweak) is now baked into PR #4329:

c2e6b38

Copying the commit message here (I think the link will stop working in case I have to rebase):

Modified version of PR #4293 by @wjakob

Modifications are:

* Backward compatibility (no ABI break), as originally under PR #4307.
* Naming: `get_python_state_dict()`, `has_pybind11_internals_capsule()`
* Report error retrieving `internals**` from capsule instead of clearing it.

Locally tested with ASAN, MSAN, TSAN, UBSAN (Google-internal toolchain).

My commit changes the code here, to report the error rather than suppressing it.

What is the rationale for suppressing it?

}

if (internals_pp) {
// We loaded builtins through python's builtins, which means that our `error_already_set`
// and `builtin_exception` may be different local classes than the ones set up in the
// initial exception translator, below, so add another for our local exception classes.
Expand All @@ -435,9 +454,7 @@ PYBIND11_NOINLINE internals &get_internals() {
(*internals_pp)->registered_exception_translators.push_front(&translate_local_exception);
#endif
} else {
if (!internals_pp) {
internals_pp = new internals *();
}
internals_pp = new internals *();
auto *&internals_ptr = *internals_pp;
internals_ptr = new internals();
#if defined(WITH_THREAD)
Expand All @@ -459,7 +476,7 @@ PYBIND11_NOINLINE internals &get_internals() {
# endif
internals_ptr->istate = tstate->interp;
#endif
builtins[id] = capsule(internals_pp);
state_dict[id] = capsule(internals_pp, id_cstr);
internals_ptr->registered_exception_translators.push_front(&translate_exception);
internals_ptr->static_property_type = make_static_property_type();
internals_ptr->default_metaclass = make_default_metaclass();
Expand Down