Skip to content

bpo-38328: Speed up the creation time of constant list and set literals. #17114

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
methane merged 12 commits into from
Nov 26, 2019
Merged

bpo-38328: Speed up the creation time of constant list and set literals. #17114

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
f556e5e
Constify and unpack constant list and set literals.
brandtbucher Nov 12, 2019
f2f77fa
Regenerate importlib.
brandtbucher Nov 12, 2019
5006646
Resize overallocated list.
brandtbucher Nov 12, 2019
bebcf00
Branch on outer op, not single op.
brandtbucher Nov 12, 2019
2af72df
Handle PyFrozenSet_New failure.
brandtbucher Nov 12, 2019
58c96cf
Avoid name reuse.
brandtbucher Nov 12, 2019
6627e15
Add NEWS entry.
brandtbucher Nov 12, 2019
d61d204
Remove random cache file.
brandtbucher Nov 12, 2019
c140a12
Catch up with master.
brandtbucher Nov 25, 2019
f405957
Update 2019-11-11-23-44-15.bpo-38328.IFrrjq.rst
methane Nov 25, 2019
5285d67
Bump threshold to enable optimization.
brandtbucher Nov 25, 2019
3dfd17c
Regenerate importlib.
brandtbucher Nov 25, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Lib/test/test_sys.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1222,7 +1222,7 @@ def get_gen(): yield 1
# list
samples = [[], [1,2,3], ['1', '2', '3']]
for sample in samples:
check(sample, vsize('Pn') + len(sample)*self.P)
check(list(sample), vsize('Pn') + len(sample)*self.P)
# sortwrapper (list)
# XXX
# cmpwrapper (list)
Expand Down
2 changes: 2 additions & 0 deletions Misc/NEWS.d/next/Core and Builtins/2019-11-11-23-44-15.bpo-38328.IFrrjq.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Sped up the creation time of constant :class:`list` and :class:`set` displays.
Patch by Brandt Bucher.
22 changes: 22 additions & 0 deletions Python/compile.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -197,6 +197,7 @@ static int compiler_visit_slice(struct compiler *, slice_ty,
expr_context_ty);

static int inplace_binop(struct compiler *, operator_ty);
static int are_all_items_const(asdl_seq *, Py_ssize_t, Py_ssize_t);
static int expr_constant(expr_ty);

static int compiler_with(struct compiler *, stmt_ty, int);
Expand Down Expand Up @@ -3655,6 +3656,27 @@ starunpack_helper(struct compiler *c, asdl_seq *elts,
{
Py_ssize_t n = asdl_seq_LEN(elts);
Py_ssize_t i, nsubitems = 0, nseen = 0;
if (n > 1 && are_all_items_const(elts, 0, n)) {
Copy link
Member

@methane methane Nov 25, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is n > 1 really good threshold? How about n > 2?

$ python3 -m timeit -s "a=frozenset((1,2))" -- "{*a}"
5000000 loops, best of 5: 63.4 nsec per loop
$ python3 -m timeit "{1,2}"
5000000 loops, best of 5: 61.5 nsec per loop

$ python3 -m timeit "[*(1,2)]"
5000000 loops, best of 5: 46.5 nsec per loop
$ python3 -m timeit "[1,2]"
5000000 loops, best of 5: 46.6 nsec per loop

Copy link
Member Author

@brandtbucher brandtbucher Nov 25, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I only chose n > 1 because it always emits fewer opcodes than before the change. If you think n > 2 is a better heuristic, I'm fine with that.

Copy link
Member Author

@brandtbucher brandtbucher Nov 25, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think your first example is only slower because it pays the price of a LOAD_NAME instead of a LOAD_CONST (just a guess, though). It's not a direct comparison, unfortunately.

Copy link
Member

@methane methane Nov 25, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fewer opcodes is not always more efficient.
BUILD_LIST_UNPACK calls PyList_New(0) and PyList_Extend(iterable). It causes over-allocation.
On the other hand, BUILD_LIST just calls PyList_New(n).

So I think we should be conservative about choosing the threshold.

Copy link
Member Author

@brandtbucher brandtbucher Nov 25, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the feedback @methane. I think this makes sense.

I've bumped the threshold to your recommended value of n > 2. Anything else?

PyObject *folded = PyTuple_New(n);
if (folded == NULL) {
return 0;
}
PyObject *val;
for (i = 0; i < n; i++) {
val = ((expr_ty)asdl_seq_GET(elts, i))->v.Constant.value;
Py_INCREF(val);
PyTuple_SET_ITEM(folded, i, val);
}
if (outer_op == BUILD_SET_UNPACK) {
Py_SETREF(folded, PyFrozenSet_New(folded));
if (folded == NULL) {
return 0;
}
}
ADDOP_LOAD_CONST_NEW(c, folded);
ADDOP_I(c, outer_op, 1);
return 1;
}
for (i = 0; i < n; i++) {
expr_ty elt = asdl_seq_GET(elts, i);
if (elt->kind == Starred_kind) {
Expand Down
284 changes: 142 additions & 142 deletions Python/importlib_external.h
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading