1 .. highlight:: c
2 
3 
4 .. _embedding:
5 
6 ***************************************
7 Embedding Python in Another Application
8 ***************************************
9 
10 The previous chapters discussed how to extend Python, that is, how to extend the
11 functionality of Python by attaching a library of C functions to it.  It is also
12 possible to do it the other way around: enrich your C/C++ application by
13 embedding Python in it.  Embedding provides your application with the ability to
14 implement some of the functionality of your application in Python rather than C
15 or C++. This can be used for many purposes; one example would be to allow users
16 to tailor the application to their needs by writing some scripts in Python.  You
17 can also use it yourself if some of the functionality can be written in Python
18 more easily.
19 
20 Embedding Python is similar to extending it, but not quite.  The difference is
21 that when you extend Python, the main program of the application is still the
22 Python interpreter, while if you embed Python, the main program may have nothing
23 to do with Python --- instead, some parts of the application occasionally call
24 the Python interpreter to run some Python code.
25 
26 So if you are embedding Python, you are providing your own main program.  One of
27 the things this main program has to do is initialize the Python interpreter.  At
28 the very least, you have to call the function :c:func:`Py_Initialize`.  There are
29 optional calls to pass command line arguments to Python.  Then later you can
30 call the interpreter from any part of the application.
31 
32 There are several different ways to call the interpreter: you can pass a string
33 containing Python statements to :c:func:`PyRun_SimpleString`, or you can pass a
34 stdio file pointer and a file name (for identification in error messages only)
35 to :c:func:`PyRun_SimpleFile`.  You can also call the lower-level operations
36 described in the previous chapters to construct and use Python objects.
37 
38 
39 .. seealso::
40 
41    :ref:`c-api-index`
42       The details of Python's C interface are given in this manual. A great deal of
43       necessary information can be found here.
44 
45 
46 .. _high-level-embedding:
47 
48 Very High Level Embedding
49 =========================
50 
51 The simplest form of embedding Python is the use of the very high level
52 interface. This interface is intended to execute a Python script without needing
53 to interact with the application directly. This can for example be used to
54 perform some operation on a file. ::
55 
56    #define PY_SSIZE_T_CLEAN
57    #include <Python.h>
58 
59    int
60    main(int argc, char *argv[])
61    {
62        wchar_t *program = Py_DecodeLocale(argv[0], NULL);
63        if (program == NULL) {
64            fprintf(stderr, "Fatal error: cannot decode argv[0]\n");
65            exit(1);
66        }
67        Py_SetProgramName(program);  /* optional but recommended */
68        Py_Initialize();
69        PyRun_SimpleString("from time import time,ctime\n"
70                           "print('Today is', ctime(time()))\n");
71        if (Py_FinalizeEx() < 0) {
72            exit(120);
73        }
74        PyMem_RawFree(program);
75        return 0;
76    }
77 
78 The :c:func:`Py_SetProgramName` function should be called before
79 :c:func:`Py_Initialize` to inform the interpreter about paths to Python run-time
80 libraries.  Next, the Python interpreter is initialized with
81 :c:func:`Py_Initialize`, followed by the execution of a hard-coded Python script
82 that prints the date and time.  Afterwards, the :c:func:`Py_FinalizeEx` call shuts
83 the interpreter down, followed by the end of the program.  In a real program,
84 you may want to get the Python script from another source, perhaps a text-editor
85 routine, a file, or a database.  Getting the Python code from a file can better
86 be done by using the :c:func:`PyRun_SimpleFile` function, which saves you the
87 trouble of allocating memory space and loading the file contents.
88 
89 
90 .. _lower-level-embedding:
91 
92 Beyond Very High Level Embedding: An overview
93 =============================================
94 
95 The high level interface gives you the ability to execute arbitrary pieces of
96 Python code from your application, but exchanging data values is quite
97 cumbersome to say the least. If you want that, you should use lower level calls.
98 At the cost of having to write more C code, you can achieve almost anything.
99 
100 It should be noted that extending Python and embedding Python is quite the same
101 activity, despite the different intent. Most topics discussed in the previous
102 chapters are still valid. To show this, consider what the extension code from
103 Python to C really does:
104 
105 #. Convert data values from Python to C,
106 
107 #. Perform a function call to a C routine using the converted values, and
108 
109 #. Convert the data values from the call from C to Python.
110 
111 When embedding Python, the interface code does:
112 
113 #. Convert data values from C to Python,
114 
115 #. Perform a function call to a Python interface routine using the converted
116    values, and
117 
118 #. Convert the data values from the call from Python to C.
119 
120 As you can see, the data conversion steps are simply swapped to accommodate the
121 different direction of the cross-language transfer. The only difference is the
122 routine that you call between both data conversions. When extending, you call a
123 C routine, when embedding, you call a Python routine.
124 
125 This chapter will not discuss how to convert data from Python to C and vice
126 versa.  Also, proper use of references and dealing with errors is assumed to be
127 understood.  Since these aspects do not differ from extending the interpreter,
128 you can refer to earlier chapters for the required information.
129 
130 
131 .. _pure-embedding:
132 
133 Pure Embedding
134 ==============
135 
136 The first program aims to execute a function in a Python script. Like in the
137 section about the very high level interface, the Python interpreter does not
138 directly interact with the application (but that will change in the next
139 section).
140 
141 The code to run a function defined in a Python script is:
142 
143 .. literalinclude:: ../includes/run-func.c
144 
145 
146 This code loads a Python script using ``argv[1]``, and calls the function named
147 in ``argv[2]``.  Its integer arguments are the other values of the ``argv``
148 array.  If you :ref:`compile and link <compiling>` this program (let's call
149 the finished executable :program:`call`), and use it to execute a Python
150 script, such as:
151 
152 .. code-block:: python
153 
154    def multiply(a,b):
155        print("Will compute", a, "times", b)
156        c = 0
157        for i in range(0, a):
158            c = c + b
159        return c
160 
161 then the result should be:
162 
163 .. code-block:: shell-session
164 
165    $ call multiply multiply 3 2
166    Will compute 3 times 2
167    Result of call: 6
168 
169 Although the program is quite large for its functionality, most of the code is
170 for data conversion between Python and C, and for error reporting.  The
171 interesting part with respect to embedding Python starts with ::
172 
173    Py_Initialize();
174    pName = PyUnicode_DecodeFSDefault(argv[1]);
175    /* Error checking of pName left out */
176    pModule = PyImport_Import(pName);
177 
178 After initializing the interpreter, the script is loaded using
179 :c:func:`PyImport_Import`.  This routine needs a Python string as its argument,
180 which is constructed using the :c:func:`PyUnicode_FromString` data conversion
181 routine. ::
182 
183    pFunc = PyObject_GetAttrString(pModule, argv[2]);
184    /* pFunc is a new reference */
185 
186    if (pFunc && PyCallable_Check(pFunc)) {
187        ...
188    }
189    Py_XDECREF(pFunc);
190 
191 Once the script is loaded, the name we're looking for is retrieved using
192 :c:func:`PyObject_GetAttrString`.  If the name exists, and the object returned is
193 callable, you can safely assume that it is a function.  The program then
194 proceeds by constructing a tuple of arguments as normal.  The call to the Python
195 function is then made with::
196 
197    pValue = PyObject_CallObject(pFunc, pArgs);
198 
199 Upon return of the function, ``pValue`` is either ``NULL`` or it contains a
200 reference to the return value of the function.  Be sure to release the reference
201 after examining the value.
202 
203 
204 .. _extending-with-embedding:
205 
206 Extending Embedded Python
207 =========================
208 
209 Until now, the embedded Python interpreter had no access to functionality from
210 the application itself.  The Python API allows this by extending the embedded
211 interpreter.  That is, the embedded interpreter gets extended with routines
212 provided by the application. While it sounds complex, it is not so bad.  Simply
213 forget for a while that the application starts the Python interpreter.  Instead,
214 consider the application to be a set of subroutines, and write some glue code
215 that gives Python access to those routines, just like you would write a normal
216 Python extension.  For example::
217 
218    static int numargs=0;
219 
220    /* Return the number of arguments of the application command line */
221    static PyObject*
222    emb_numargs(PyObject *self, PyObject *args)
223    {
224        if(!PyArg_ParseTuple(args, ":numargs"))
225            return NULL;
226        return PyLong_FromLong(numargs);
227    }
228 
229    static PyMethodDef EmbMethods[] = {
230        {"numargs", emb_numargs, METH_VARARGS,
231         "Return the number of arguments received by the process."},
232        {NULL, NULL, 0, NULL}
233    };
234 
235    static PyModuleDef EmbModule = {
236        PyModuleDef_HEAD_INIT, "emb", NULL, -1, EmbMethods,
237        NULL, NULL, NULL, NULL
238    };
239 
240    static PyObject*
241    PyInit_emb(void)
242    {
243        return PyModule_Create(&EmbModule);
244    }
245 
246 Insert the above code just above the :c:func:`main` function. Also, insert the
247 following two statements before the call to :c:func:`Py_Initialize`::
248 
249    numargs = argc;
250    PyImport_AppendInittab("emb", &PyInit_emb);
251 
252 These two lines initialize the ``numargs`` variable, and make the
253 :func:`emb.numargs` function accessible to the embedded Python interpreter.
254 With these extensions, the Python script can do things like
255 
256 .. code-block:: python
257 
258    import emb
259    print("Number of arguments", emb.numargs())
260 
261 In a real application, the methods will expose an API of the application to
262 Python.
263 
264 .. TODO: threads, code examples do not really behave well if errors happen
265    (what to watch out for)
266 
267 
268 .. _embeddingincplusplus:
269 
270 Embedding Python in C++
271 =======================
272 
273 It is also possible to embed Python in a C++ program; precisely how this is done
274 will depend on the details of the C++ system used; in general you will need to
275 write the main program in C++, and use the C++ compiler to compile and link your
276 program.  There is no need to recompile Python itself using C++.
277 
278 
279 .. _compiling:
280 
281 Compiling and Linking under Unix-like systems
282 =============================================
283 
284 It is not necessarily trivial to find the right flags to pass to your
285 compiler (and linker) in order to embed the Python interpreter into your
286 application, particularly because Python needs to load library modules
287 implemented as C dynamic extensions (:file:`.so` files) linked against
288 it.
289 
290 To find out the required compiler and linker flags, you can execute the
291 :file:`python{X.Y}-config` script which is generated as part of the
292 installation process (a :file:`python3-config` script may also be
293 available).  This script has several options, of which the following will
294 be directly useful to you:
295 
296 * ``pythonX.Y-config --cflags`` will give you the recommended flags when
297   compiling:
298 
299   .. code-block:: shell-session
300 
301      $ /opt/bin/python3.11-config --cflags
302      -I/opt/include/python3.11 -I/opt/include/python3.11 -Wsign-compare  -DNDEBUG -g -fwrapv -O3 -Wall
303 
304 * ``pythonX.Y-config --ldflags --embed`` will give you the recommended flags
305   when linking:
306 
307   .. code-block:: shell-session
308 
309      $ /opt/bin/python3.11-config --ldflags --embed
310      -L/opt/lib/python3.11/config-3.11-x86_64-linux-gnu -L/opt/lib -lpython3.11 -lpthread -ldl  -lutil -lm
311 
312 .. note::
313    To avoid confusion between several Python installations (and especially
314    between the system Python and your own compiled Python), it is recommended
315    that you use the absolute path to :file:`python{X.Y}-config`, as in the above
316    example.
317 
318 If this procedure doesn't work for you (it is not guaranteed to work for
319 all Unix-like platforms; however, we welcome :ref:`bug reports <reporting-bugs>`)
320 you will have to read your system's documentation about dynamic linking and/or
321 examine Python's :file:`Makefile` (use :func:`sysconfig.get_makefile_filename`
322 to find its location) and compilation
323 options.  In this case, the :mod:`sysconfig` module is a useful tool to
324 programmatically extract the configuration values that you will want to
325 combine together.  For example:
326 
327 .. code-block:: pycon
328 
329    >>> import sysconfig
330    >>> sysconfig.get_config_var('LIBS')
331    '-lpthread -ldl  -lutil'
332    >>> sysconfig.get_config_var('LINKFORSHARED')
333    '-Xlinker -export-dynamic'
334 
335 
336 .. XXX similar documentation for Windows missing
337