what is python c-api

API used by C and C++ programmers who want to write extension modules or embed Python

what is Py_Initialize

a python c-api, not a boost api,

Initialize the Python interpreter. In an application embedding Python, this should be called before using any other Python/C API functions; see Before Python Initialization for the few exceptions. This initializes the table of loaded modules (sys.modules), and creates the fundamental modules builtins, main and sys. It also initializes the module search path (sys.path). It does not set sys.argv; use PySys_SetArgvEx() for that. This is a no-op when called for a second time (without calling Py_FinalizeEx() first). There is no return value; it is a fatal error if the initialization fails.

what is scoped_interpreter

• initialize_interpreter:

Initialize the Python interpreter. No other pybind11 or CPython API functions can be called before this is done; with the exception of PYBIND11_EMBEDDED_MODULE. The optional parameter can be used to skip the registration of signal handlers (see the Python documentation for details). Calling this function again after the interpreter has already been initialized is a fatal error.

• finalize_interpreter:

Shut down the Python interpreter. No pybind11 or CPython API functions can be called after this. In addition, pybind11 objects must not outlive the interpreter:

• scoped_interpreter:

Scope guard version of initialize_interpreter() and finalize_interpreter(). This a move-only guard and only a single instance can exist.

what the difference between Py_Initialize and scoped_interpreter, can we switch between them

• if we call Py_Initialize twice, the second call will do nothing
• if we cann scoped_interpreter twice, program will crash
• yes we can switch between them, we can demostrate this

when to use Py_Initialize or scoped_interpreter

• we only need to use them in embed python when no python interpreter is started
• we don’t need and shouldn’t use them in python call c++ case

what if we use Py_Initialize or scoped_interpreter in python call c++ condition

• Py_Initialize is ok because it will do nothing
• scoped_interpreter will crash

what is GIL

• Global Interpreter Lock

In python, you can only execute one thread at a time as it has GIL.

Python has a “reference-counter” for memory management, without gil, reference-counter will be broken in race condition

how to manage GIL in boost py?

by calling python api:

• PyEval_SaveThread

Release the global interpreter lock (if it has been created) and reset the thread state to NULL, returning the previous thread state (which is not NULL). If the lock has been created, the current thread must have acquired it.

• PyEval_RestoreThread

Acquire the global interpreter lock (if it has been created) and set the thread state to tstate, which must not be NULL. If the lock has been created, the current thread must not have acquired it, otherwise deadlock ensues.

• PyGILState_Ensure

Ensure that the current thread is ready to call the Python C API regardless of the current state of Python, or of the global interpreter lock

• PyGILState_Release

Release any resources previously acquired

when threads are created from C, they don’t have the global interpreter lock, nor is there a thread state data structure for them. Such threads must bootstrap themselves into existence, by first creating a thread state data structure, then acquiring the lock, and finally storing their thread state pointer, before they can start using the Python/C API.

so when we switch from cpp to python, we need ensure/release pair, when we switch from python to cpp, we need save/restore pair

better way to manage GIL in boost py?

with gil and without gil, we can demostrate this

what the difference if we don’t manage GIL?

slower in multi-thread python program, we can domostrate this

what will happen if we get confused and mix things up?

our boostpy example crash, we cannot call python without gil

how to manage GIL in pybind11? what is the difference between pybind11 style and boostpy style

py::gil_scoped_release and py::gil_scoped_acquire, actually without_gil and with_gil

can we use boost py style in pybind11 to manage GIL?

yes, we can demostrate this, vice versa