Skip to content

bpo-43605: Improve the documentation to exec() and eval() #25039

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
congma wants to merge 3 commits into
base: main
Choose a base branch
from
Open

bpo-43605: Improve the documentation to exec() and eval() #25039

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ecd5e7b
bpo-43605: Improve the documentation to exec() and eval()
congma Mar 27, 2021
4ffc9cd
Further revise eval/exec documentations for consistency and accuracy.
congma Mar 29, 2021
6504b9c
Fix invalid RST syntax.
congma Mar 29, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
133 changes: 76 additions & 57 deletions Doc/library/functions.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -496,44 +496,38 @@ are always available. They are listed here in alphabetical order.

.. function:: eval(expression[, globals[, locals]])

The arguments are a string and optional globals and locals. If provided,
*globals* must be a dictionary. If provided, *locals* can be any mapping
object.

The *expression* argument is parsed and evaluated as a Python expression
(technically speaking, a condition list) using the *globals* and *locals*
dictionaries as global and local namespace. If the *globals* dictionary is
present and does not contain a value for the key ``__builtins__``, a
reference to the dictionary of the built-in module :mod:`builtins` is
inserted under that key before *expression* is parsed. That way you can
control what builtins are available to the executed code by inserting your
own ``__builtins__`` dictionary into *globals* before passing it to
:func:`eval`. If the *locals* dictionary is omitted it defaults to the
*globals* dictionary. If both dictionaries are omitted, the expression is
executed with the *globals* and *locals* in the environment where
:func:`eval` is called. Note, *eval()* does not have access to the
:term:`nested scopes <nested scope>` (non-locals) in the enclosing
environment.

The return value is the result of
the evaluated expression. Syntax errors are reported as exceptions. Example:
This function supports the dynamic evaluation of Python expression. The
first argument can be a string or a code object. The optional arguments
Copy link
Member

@JelleZijlstra JelleZijlstra Apr 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe link this to the documentation for code objects.

specify the globals and locals respectively. If provided, *globals* must be
a dictionary. If provided, *locals* can be any mapping object.
Copy link
Member

@Fidget-Spinner Fidget-Spinner Mar 27, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think locals can be any mapping object. Eg. collections.Counter() is a mapping object (isinstance(collections.Counter('a'), collections.abc.Mapping) is True), but it cannot be used for eval:

>>> Counter("abcd")
Counter({'a': 1, 'b': 1, 'c': 1, 'd': 1})
>>> eval("print(a)", {}, Counter("abcd"))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 1, in <module>
TypeError: 'int' object is not callable
>>> eval("print(a)", {}, {'a':1})
1

Surprisingly, git blame says this line has been around since this paragraph's inception 14 years ago.

Suggested change
specify the globals and locals respectively. If provided, *globals* must be
a dictionary. If provided, *locals* can be any mapping object.
specify the globals and locals respectively. If provided, *globals*
and *locals* must be dictionaries.

Copy link
Author

@congma congma Mar 27, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What?? In the source for eval/exec in Python/bltinmodule.c, the locals are checked by PyMapping_Check() while globals checked by PyDict_Check(). I'm not sure why the exception was about "'int' object is not callable" -- it seems the name a did get resolved in the locals through the Counter object, but somehow there was an action to the effect of 1(). This might be a bug! Right now I don't have enough spoons to track it down, but I should be verifying it and maybe submitting a bug report soon. Edit: working as intended, but in a tricky way: see my reply below.

Copy link
Author

@congma congma Mar 27, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This works as intended:

>>> ct = Counter("abcd")
>>> eval("a", {}, ct)
1

But not this:

>>> eval("print(a)", {}, ct)

Expected return value of eval is None (return value of print function), but the cryptic exception is raised.

Copy link
Author

@congma congma Mar 27, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, this:

>>> eval("print(1)", {}, ct)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 1, in <module>
TypeError: 'int' object is not callable

Copy link
Author

@congma congma Mar 27, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see! The locals have the highest priority in name resolution. The Counter instance (ct in the above example) behaves as if any key is in it (with initial value 0). When the machinery underlying eval does its work, it first looks up the name "print" in the locals given to it. Unfortunately, as a result of how Counter behaves, it gets what it wishes for: the count for the key "print" in the counter ct, which is the int instance 0. Then 0 gets called as the function. The evaluation boils down to 0(1) which raises TypeError.

Copy link
Member

@Fidget-Spinner Fidget-Spinner Mar 28, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome! Thank you for investigating this! I learnt a lot :).

I forgot that Counter sort of behaves like a defaultdict and returns 0 for keys it doesn't have.

Copy link
Author

@congma congma Mar 28, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, me too. This is the weirdest thing I've seen in Python in a while.


In the most common case, the *expression* argument is a string, and it is
Copy link
Member

@JelleZijlstra JelleZijlstra Apr 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
In the most common case, the *expression* argument is a string, and it is
When the *expression* argument is a string, it is

Tighten the wording

parsed and evaluated as a Python expression (see the section
":ref:`expression-input`" in the Language Reference). The leading and
trailing spaces, tabs, and newlines are stripped.

The evaluation is performed using the *globals* and *locals* dictionaries as
the global and local namespaces. If the *globals* dictionary is present and
does not contain a value for the key ``"__builtins__"``, a reference to the
dictionary of the built-in module :mod:`builtins` is inserted under that key
before *expression* is parsed. That way, you can control what builtins are
available to the executed code by inserting your own ``"__builtins__"``
dictionary into *globals* before passing it to :func:`eval`. If the
*locals* dictionary is omitted, it defaults to the *globals* dictionary. If
both dictionaries are omitted, the expression is executed with the globals
and locals in the environment where :func:`eval` is called.

The return value is the result of the evaluated expression. Syntax errors
are reported as exceptions. For example:

>>> x = 1
>>> eval('x+1')
2

This function can also be used to execute arbitrary code objects (such as
those created by :func:`compile`). In this case pass a code object instead
of a string. If the code object has been compiled with ``'exec'`` as the
*mode* argument, :func:`eval`\'s return value will be ``None``.

Hints: dynamic execution of statements is supported by the :func:`exec`
function. The :func:`globals` and :func:`locals` functions
returns the current global and local dictionary, respectively, which may be
useful to pass around for use by :func:`eval` or :func:`exec`.

If the given source is a string, then leading and trailing spaces and tabs
are stripped.
The :func:`eval` function can also be used to execute code objects, such as
those created by :func:`compile`. In this case, pass a code object instead
of a string as the first argument. If the code object has been compiled
with ``'exec'`` as the *mode* argument, the return value will be ``None``.

See :func:`ast.literal_eval` for a function that can safely evaluate strings
with expressions containing only literals.
Expand All @@ -543,36 +537,58 @@ are always available. They are listed here in alphabetical order.
Raises an :ref:`auditing event <auditing>` ``exec`` with the code object
as the argument. Code compilation events may also be raised.

.. _eval_limitation_dynamic:
.. note::

Dynamic evaluation at run-time is not equivalent to embedding the
expression in a Python program and having it compiled as a part of the
whole program. In particular, :func:`eval` does not have access to the
:term:`nested scopes <nested scope>` (non-locals) in the enclosing
environment (see the section ":ref:`dynamic-features`" in the Language
Reference chapter on name binding). Of note are some expressions, such
as :term:`lambdas <lambda>`, :ref:`comprehensions <comprehensions>`, and
:term:`generator expressions <generator expression>`, which create an
inner scope of their own. The interaction between these expressions and
:func:`eval` can be explicitly controlled by the parameters *globals* and
*locals* in the aforementioned manner.

The built-in functions :func:`globals` and :func:`locals` return the
current global and local dictionary, respectively, which may be useful to
pass around for use as the second and third argument to :func:`eval`.

Dynamic execution of *statements* is supported by the :func:`exec`
function (see below).


.. index:: builtin: exec

.. function:: exec(object[, globals[, locals]])

This function supports dynamic execution of Python code. *object* must be
either a string or a code object. If it is a string, the string is parsed as
a suite of Python statements which is then executed (unless a syntax error
occurs). [#]_ If it is a code object, it is simply executed. In all cases,
the code that's executed is expected to be valid as file input (see the
section "File input" in the Reference Manual). Be aware that the
:keyword:`nonlocal`, :keyword:`yield`, and :keyword:`return`
statements may not be used outside of
function definitions even within the context of code passed to the
:func:`exec` function. The return value is ``None``.

In all cases, if the optional parts are omitted, the code is executed in the
current scope. If only *globals* is provided, it must be a dictionary
(and not a subclass of dictionary), which
will be used for both the global and the local variables. If *globals* and
*locals* are given, they are used for the global and local variables,
respectively. If provided, *locals* can be any mapping object. Remember
that at module level, globals and locals are the same dictionary. If exec
gets two separate objects as *globals* and *locals*, the code will be
executed as if it were embedded in a class definition.
either a string or a code object. If it is a string, the string is parsed
as a suite of Python statements which is then executed (unless a syntax
error occurs). [#]_ If it is a code object, it is simply executed. In all
cases, the code that's executed is expected to be valid as file input (see
the section ":ref:`file-input`" in the Language Reference). Be aware that
the :keyword:`nonlocal`, :keyword:`yield`, and :keyword:`return` statements
may not be used outside of function definitions even within the context of
code passed to the :func:`exec` function. The return value is ``None``.

In all cases, if the optional arguments are omitted, the code is executed in
the current scope. If only *globals* is provided, it must be a dictionary,
which will be used for both the global and the local variables. If
*globals* and *locals* are given, they are used for the global and local
variables, respectively. If provided, *locals* can be any mapping object.
Remember that at module level, globals and locals are the same dictionary.
If exec gets two separate objects as *globals* and *locals*, the code will
be executed as if it were embedded in a class definition.

If the *globals* dictionary does not contain a value for the key
``__builtins__``, a reference to the dictionary of the built-in module
``"__builtins__"``, a reference to the dictionary of the built-in module
:mod:`builtins` is inserted under that key. That way you can control what
builtins are available to the executed code by inserting your own
``__builtins__`` dictionary into *globals* before passing it to :func:`exec`.
``"__builtins__"`` dictionary into *globals* before passing it to
:func:`exec`.

.. audit-event:: exec code_object exec

Expand All @@ -581,9 +597,12 @@ are always available. They are listed here in alphabetical order.

.. note::

The built-in functions :func:`globals` and :func:`locals` return the current
global and local dictionary, respectively, which may be useful to pass around
for use as the second and third argument to :func:`exec`.
Like :func:`eval`, :func:`exec` is :ref:`limited by constraints
particular to dynamic code execution and namespaces
<eval_limitation_dynamic>`. The built-in functions :func:`globals` and
:func:`locals` return the current global and local dictionary,
respectively, which may be useful to pass around for use as the second
and third argument to :func:`exec`.

.. note::

Expand Down