Skip to content

Commit 0f005ad

Browse files
bashtagebashtage
authored
Merge pull request #244 from bashtage/qa-4
DOC: Additional QA results
2 parents 5ca2cca + 583e3df commit 0f005ad

File tree

9 files changed

+592
-419
lines changed

9 files changed

+592
-419
lines changed

‎doc/source/performance.rst‎

Lines changed: 59 additions & 54 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,8 @@ Recommendation
77
**************
88
The recommended generator for single use is :class:`~randomgen.pcg64.PCG64DXSM`
99
although :class:`~randomgen.sfc64.SFC64` and :class:`~randomgen.xoshiro256.Xoshiro256`
10-
are both excellent alternatives.
10+
are both excellent alternatives. :class:`~randomgen.romu.Romu` is a newer generator that
11+
is also very fast.
1112

1213
For very large scale
1314
applications -- requiring 1,000+ streams --
@@ -37,33 +38,35 @@ it requires 2 32-bit values to equal the output of the faster generators.
3738
:header: Bit Gen,Uint32,Uint64,Uniform,Expon,Normal,Gamma
3839
:widths: 30,10,10,10,10,10,10
3940

40-
SFC64,2.3,2.5,2.4,5.1,10.6,18.7
41-
Xoshiro256,2.5,2.5,2.5,5.2,9.7,19.1
42-
PCG64DXSM,2.3,2.8,3.0,5.3,11.0,20.7
43-
JSF,2.3,3.0,3.0,5.7,10.2,20.0
44-
EFIIX64,2.5,3.0,3.0,5.5,10.7,20.8
45-
PCG64,2.3,3.1,3.1,5.8,11.2,21.4
46-
Xoshiro512,2.6,3.5,3.3,5.8,10.3,20.3
47-
SFMT,2.9,3.3,3.1,6.3,10.9,20.8
48-
LXM,2.6,3.5,3.5,6.3,11.3,21.9
49-
PCG64(variant="dxsm-128"),2.8,3.3,3.5,6.1,12.5,24.3
50-
DSFMT,3.0,4.2,2.7,7.0,12.1,21.6
51-
MT64,2.8,4.0,4.2,6.8,12.8,23.7
52-
JSF32,3.0,4.3,4.3,6.9,11.2,22.9
53-
"Philox(n=2, w=64)",3.3,4.7,5.1,8.1,14.7,27.4
54-
Philox,3.9,5.9,6.0,8.8,13.6,26.9
55-
AESCounter,4.4,6.0,5.9,9.1,14.5,27.4
56-
MT19937,3.8,6.2,7.2,9.2,14.8,28.7
57-
"ThreeFry(n=2, w=64)",4.1,6.5,6.9,9.4,15.7,29.8
58-
NumPy,3.0,4.7,5.8,16.7,20.8,40.2
59-
HC128,4.1,7.2,7.2,10.5,16.6,31.6
60-
"Philox(n=4, w=32)",4.2,7.6,8.5,10.7,16.5,32.6
61-
SPECK128,5.4,8.1,9.6,11.3,17.0,33.3
62-
ThreeFry,5.8,9.1,9.3,12.0,16.7,34.9
63-
ChaCha(rounds=8),6.7,10.2,10.2,13.2,18.0,36.4
64-
ChaCha,9.6,16.4,16.4,19.6,24.3,48.9
65-
"ThreeFry(n=4, w=32)",9.0,16.3,17.6,20.7,24.4,53.4
66-
RDRAND,129.9,133.1,130.7,139.0,140.8,298.6
41+
"Romu(variant=""trio"")",2.3,2.1,2.0,5.2,9.3,18.2
42+
SFC64,2.3,2.5,2.4,5.1,9.6,18.7
43+
Romu,2.5,2.5,2.4,5.0,9.5,18.8
44+
Xoshiro256,2.5,2.5,2.5,5.2,9.8,19.2
45+
PCG64DXSM,2.3,2.9,3.0,5.3,11.0,20.7
46+
JSF,2.4,3.0,3.0,5.8,10.2,20.0
47+
EFIIX64,2.5,3.0,3.0,5.4,10.5,20.4
48+
PCG64,2.3,3.1,3.1,5.9,11.3,21.4
49+
Xoshiro512,2.7,3.5,3.3,5.8,10.3,20.3
50+
SFMT,2.9,3.3,3.1,6.3,11.0,20.9
51+
LXM,2.6,3.5,3.5,6.3,11.3,21.9
52+
"PCG64(variant=""dxsm-128"")",2.8,3.4,3.5,6.1,12.5,23.1
53+
DSFMT,3.0,4.2,2.7,7.0,12.2,21.7
54+
MT64,2.8,4.0,4.2,6.9,12.8,23.7
55+
JSF32,3.0,4.3,4.3,6.9,11.2,22.9
56+
"Philox(n=2, w=64)",3.2,4.7,5.1,8.1,14.7,27.4
57+
Philox,4.0,5.9,6.1,8.8,13.6,27.0
58+
AESCounter,4.4,6.0,5.7,9.1,14.4,27.4
59+
MT19937,3.8,6.3,7.2,9.2,14.8,28.8
60+
"ThreeFry(n=2, w=64)",4.1,6.5,6.9,9.5,15.7,30.6
61+
NumPy,3.0,4.6,5.8,14.4,20.1,39.8
62+
HC128,4.1,7.2,7.2,10.5,16.6,31.5
63+
"Philox(n=4, w=32)",4.2,7.6,8.7,10.8,16.5,32.6
64+
SPECK128,5.4,8.1,9.7,11.4,17.0,33.4
65+
ThreeFry,5.9,9.1,9.3,12.0,16.7,34.9
66+
ChaCha(rounds=8),6.7,10.4,10.3,13.2,18.1,36.4
67+
ChaCha,9.7,16.6,16.4,19.6,24.4,49.0
68+
"ThreeFry(n=4, w=32)",9.0,16.8,17.5,20.7,24.3,54.1
69+
RDRAND,129.5,129.9,129.6,136.9,139.6,287.4
6770

6871
The next table presents the performance relative to NumPy's ``RandomState`` in
6972
percentage. The overall performance is computed using a geometric mean.
@@ -72,32 +75,34 @@ percentage. The overall performance is computed using a geometric mean.
7275
:header: Bit Gen,Uint32,Uint64,Uniform,Expon,Normal,Gamma,Overall
7376
:widths: 30,10,10,10,10,10,10,10
7477

75-
SFC64,128,190,246,283,210,211,205
76-
Xoshiro256,120,186,229,280,206,207,198
77-
JSF,126,155,193,251,197,198,183
78-
EFIIX64,120,154,193,265,191,194,181
79-
PCG64DXSM,128,163,191,248,164,192,177
80-
PCG64,129,149,185,247,179,185,175
81-
Xoshiro512,111,132,177,251,195,195,171
82-
SFMT,104,142,186,229,183,190,167
83-
LXM,114,135,165,228,177,181,163
84-
PCG64(variant="dxsm-128"),105,139,165,235,161,172,158
85-
DSFMT,100,110,214,207,166,184,157
86-
JSF32,99,108,134,209,179,173,145
87-
MT64,105,117,137,211,157,167,145
88-
"Philox(n=2, w=64)",89,98,114,177,137,145,123
89-
Philox,75,79,95,165,148,147,112
90-
AESCounter,67,77,102,159,139,144,109
91-
MT19937,77,74,80,157,135,138,105
92-
"ThreeFry(n=2, w=64)",72,71,84,153,128,133,102
93-
HC128,72,65,80,137,121,125,96
94-
"Philox(n=4, w=32)",71,61,68,134,122,122,92
95-
SPECK128,55,57,60,127,118,119,83
96-
ThreeFry,51,53,63,120,120,114,81
97-
ChaCha(rounds=8),45,46,56,109,111,109,73
98-
ChaCha,31,28,35,74,83,81,50
99-
"ThreeFry(n=4, w=32)",33,28,33,70,83,76,49
100-
RDRAND,2,3,4,10,14,13,6
78+
"Romu(variant=""trio"")",128,219,282,278,215,219,217
79+
SFC64,129,189,244,283,210,213,205
80+
Romu,118,185,242,285,212,212,202
81+
Xoshiro256,119,185,229,279,205,208,198
82+
PCG64DXSM,128,163,190,271,182,192,183
83+
JSF,126,155,193,250,197,199,182
84+
EFIIX64,120,154,193,264,190,195,181
85+
PCG64,128,149,184,246,178,186,175
86+
Xoshiro512,111,131,176,249,195,196,170
87+
SFMT,104,143,185,228,183,190,167
88+
LXM,114,135,164,227,177,182,162
89+
"PCG64(variant=""dxsm-128"")",105,138,165,234,160,172,158
90+
DSFMT,100,110,213,206,165,184,156
91+
MT64,104,117,136,210,156,168,145
92+
JSF32,99,108,134,208,178,174,145
93+
"Philox(n=2, w=64)",91,98,114,177,136,145,124
94+
Philox,75,79,95,163,147,147,112
95+
AESCounter,67,77,102,158,139,145,109
96+
MT19937,78,74,80,156,135,138,105
97+
"ThreeFry(n=2, w=64)",72,71,84,152,128,130,101
98+
HC128,72,64,80,136,121,126,96
99+
"Philox(n=4, w=32)",71,61,66,134,122,122,91
100+
SPECK128,55,57,59,127,118,119,83
101+
ThreeFry,51,51,62,120,120,114,80
102+
ChaCha(rounds=8),45,45,56,109,111,109,73
103+
ChaCha,30,28,35,73,82,81,49
104+
"ThreeFry(n=4, w=32)",33,28,33,69,82,73,48
105+
RDRAND,2,4,4,10,14,14,7
101106

102107
.. note::
103108

‎doc/source/test-results.txt‎

Lines changed: 55 additions & 55 deletions
Original file line numberDiff line numberDiff line change
@@ -1,55 +1,55 @@
1-
+------------------------------+-----------------------------------+-----------------------+
2-
| Method | Seed Sequence | Jumped |
3-
+------------------------------+-----------+-----------+-----------+-----------+-----------+
4-
| Streams | 1 | 4 | 8196 | 4 | 8196 |
5-
+==============================+===========+===========+===========+===========+===========+
6-
| AESCounter | 4TB | 4TB | 4TB | 4TB | 4TB |
7-
+------------------------------+-----------+-----------+-----------+-----------+-----------+
8-
| ChaCha(rounds=20) | 4TB | 4TB | 4TB | 4TB | 4TB |
9-
+------------------------------+-----------+-----------+-----------+-----------+-----------+
10-
| ChaCha(rounds=8) | 4TB | 4TB | 4TB | 4TB | 4TB |
11-
+------------------------------+-----------+-----------+-----------+-----------+-----------+
12-
| DSFMT⁴ | 4TB | FAIL¹ | 4TB | FAIL¹ | FAIL¹ |
13-
+------------------------------+-----------+-----------+-----------+-----------+-----------+
14-
| EFIIX64 | 1TB | 1TB | 1TB | -- | -- |
15-
+------------------------------+-----------+-----------+-----------+-----------+-----------+
16-
| HC128 | 4TB | 4TB | 4TB | -- | -- |
17-
+------------------------------+-----------+-----------+-----------+-----------+-----------+
18-
| JSF | 4TB | 4TB | 4TB | -- | -- |
19-
+------------------------------+-----------+-----------+-----------+-----------+-----------+
20-
| JSF(seed_size=3) | 4TB | 4TB | 4TB | -- | -- |
21-
+------------------------------+-----------+-----------+-----------+-----------+-----------+
22-
| LCG128Mix(output=upper) | 4TB | 4TB | 4TB | 4TB | 4TB |
23-
+------------------------------+-----------+-----------+-----------+-----------+-----------+
24-
| LXM | 4TB | 4TB | 4TB | 4TB | 4TB |
25-
+------------------------------+-----------+-----------+-----------+-----------+-----------+
26-
| MT19937⁴,⁵ | 4TB | FAIL¹ | 4TB | FAIL¹ | 4TB |
27-
+------------------------------+-----------+-----------+-----------+-----------+-----------+
28-
| PCG64DXSM² | 4TB | 1TB | 4TB | 1TB | 1TB |
29-
+------------------------------+-----------+-----------+-----------+-----------+-----------+
30-
| PCG64(variant=dxsm-128) | 1TB | 1TB | 4TB | 4TB | 1TB |
31-
+------------------------------+-----------+-----------+-----------+-----------+-----------+
32-
| PCG64⁵ | 4TB | 4TB | 1TB | 4TB | 4TB |
33-
+------------------------------+-----------+-----------+-----------+-----------+-----------+
34-
| Philox⁵ | 4TB | 4TB | 4TB | 4TB | 4TB |
35-
+------------------------------+-----------+-----------+-----------+-----------+-----------+
36-
| Romu | 128GB | 128GB | 128GB | -- | -- |
37-
+------------------------------+-----------+-----------+-----------+-----------+-----------+
38-
| Romu(variant=trio) | 128GB | 128GB | 128GB | -- | -- |
39-
+------------------------------+-----------+-----------+-----------+-----------+-----------+
40-
| SFC64⁵ | 1TB | 4TB | 4TB | -- | -- |
41-
+------------------------------+-----------+-----------+-----------+-----------+-----------+
42-
| SFC64(k=3394385948627484371) | 1TB | 1TB | 1TB | -- | -- |
43-
+------------------------------+-----------+-----------+-----------+-----------+-----------+
44-
| SFC64(k=Weyl)³ | 4TB | 1TB | 1TB | -- | -- |
45-
+------------------------------+-----------+-----------+-----------+-----------+-----------+
46-
| SFMT⁴ | 1TB | FAIL¹ | 4TB | FAIL¹ | 1TB |
47-
+------------------------------+-----------+-----------+-----------+-----------+-----------+
48-
| SPECK128 | 4TB | 1TB | 4TB | 1TB | 4TB |
49-
+------------------------------+-----------+-----------+-----------+-----------+-----------+
50-
| ThreeFry | 4TB | 4TB | 4TB | 4TB | 4TB |
51-
+------------------------------+-----------+-----------+-----------+-----------+-----------+
52-
| Xoshiro256 | 4TB | 4TB | 4TB | 4TB | 4TB |
53-
+------------------------------+-----------+-----------+-----------+-----------+-----------+
54-
| Xoshiro512 | 4TB | 4TB | 1TB | 4TB | 4TB |
55-
+------------------------------+-----------+-----------+-----------+-----------+-----------+
1+
+------------------------------+-----------------------------------------------+-------------------------------+
2+
| Method | Seed Sequence | Jumped |
3+
+------------------------------+---------------+---------------+---------------+---------------+---------------+
4+
| Streams | 1 | 4 | 8196 | 4 | 8196 |
5+
+==============================+===============+===============+===============+===============+===============+
6+
| AESCounter | 4TB | 4TB | 4TB | 4TB | 4TB |
7+
+------------------------------+---------------+---------------+---------------+---------------+---------------+
8+
| ChaCha(rounds=20) | 4TB | 4TB | 4TB | 4TB | 4TB |
9+
+------------------------------+---------------+---------------+---------------+---------------+---------------+
10+
| ChaCha(rounds=8) | 4TB | 4TB | 4TB | 4TB | 4TB |
11+
+------------------------------+---------------+---------------+---------------+---------------+---------------+
12+
| DSFMT⁴ | 4TB |FAIL at 64 GB¹ | 4TB |FAIL at 64 GB¹ |FAIL at 64 GB¹ |
13+
+------------------------------+---------------+---------------+---------------+---------------+---------------+
14+
| EFIIX64 | 4TB | 4TB | 4TB | -- | -- |
15+
+------------------------------+---------------+---------------+---------------+---------------+---------------+
16+
| HC128 | 4TB | 4TB | 4TB | -- | -- |
17+
+------------------------------+---------------+---------------+---------------+---------------+---------------+
18+
| JSF | 4TB | 4TB | 4TB | -- | -- |
19+
+------------------------------+---------------+---------------+---------------+---------------+---------------+
20+
| JSF(seed_size=3) | 4TB | 4TB | 4TB | -- | -- |
21+
+------------------------------+---------------+---------------+---------------+---------------+---------------+
22+
| LCG128Mix(output=upper) | 4TB | 4TB | 4TB | 4TB | 4TB |
23+
+------------------------------+---------------+---------------+---------------+---------------+---------------+
24+
| LXM | 4TB | 4TB | 4TB | 4TB | 4TB |
25+
+------------------------------+---------------+---------------+---------------+---------------+---------------+
26+
| MT19937⁴,⁵ | 4TB |FAIL at 64 GB¹ | 4TB |FAIL at 64 GB¹ | 4TB |
27+
+------------------------------+---------------+---------------+---------------+---------------+---------------+
28+
| PCG64DXSM² | 4TB | 4TB | 4TB | 4TB | 4TB |
29+
+------------------------------+---------------+---------------+---------------+---------------+---------------+
30+
| PCG64(variant=dxsm-128) | 4TB | 4TB | 4TB | 4TB | 4TB |
31+
+------------------------------+---------------+---------------+---------------+---------------+---------------+
32+
| PCG64⁵ | 4TB | 4TB | 4TB | 4TB | 4TB |
33+
+------------------------------+---------------+---------------+---------------+---------------+---------------+
34+
| Philox⁵ | 4TB | 4TB | 4TB | 4TB | 4TB |
35+
+------------------------------+---------------+---------------+---------------+---------------+---------------+
36+
| Romu | 4TB | 4TB | 4TB | -- | -- |
37+
+------------------------------+---------------+---------------+---------------+---------------+---------------+
38+
| Romu(variant=trio) | 4TB | 4TB | 4TB | -- | -- |
39+
+------------------------------+---------------+---------------+---------------+---------------+---------------+
40+
| SFC64⁵ | 4TB | 4TB | 4TB | -- | -- |
41+
+------------------------------+---------------+---------------+---------------+---------------+---------------+
42+
| SFC64(k=3394385948627484371) | 4TB | 4TB | 4TB | -- | -- |
43+
+------------------------------+---------------+---------------+---------------+---------------+---------------+
44+
| SFC64(k=Weyl)³ | 4TB | 4TB | 4TB | -- | -- |
45+
+------------------------------+---------------+---------------+---------------+---------------+---------------+
46+
| SFMT⁴ | 4TB |FAIL at 64 GB¹ | 4TB |FAIL at 64 GB¹ | FAIL at 4 TB¹ |
47+
+------------------------------+---------------+---------------+---------------+---------------+---------------+
48+
| SPECK128 | 4TB | 4TB | 4TB | 4TB | 4TB |
49+
+------------------------------+---------------+---------------+---------------+---------------+---------------+
50+
| ThreeFry | 4TB | 4TB | 4TB | 4TB | 4TB |
51+
+------------------------------+---------------+---------------+---------------+---------------+---------------+
52+
| Xoshiro256 | 4TB | 4TB | 4TB | 4TB | 4TB |
53+
+------------------------------+---------------+---------------+---------------+---------------+---------------+
54+
| Xoshiro512 | 4TB | 4TB | 4TB | 4TB | 4TB |
55+
+------------------------------+---------------+---------------+---------------+---------------+---------------+

‎doc/source/testing.rst‎

Lines changed: 42 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,12 @@ Quality Assurance
33
=================
44

55
A values below are the maximum output size where a bit generator or sequence of bit generators
6-
has passed PractRand_.ꭞ A -- indicates that configuration is not relevant. Failures are marked
6+
has passed PractRand_. A -- indicates that configuration is not relevant. Failures are marked
77
with FAIL. Most bit generators were only tested in their default configuration.
88
Non-default configurations are indicated by listing the keyword arguments to the bit generator.
9+
Two sets of tests were performed. The first tested all configurations using 128GB of data using
10+
PractRand's extended set of tests and additional bit folding. The second set of tests used
11+
4TB of data wit the standard set of tests and folding.
912

1013
All bit generators have been tested using the same :class:`~numpy.random.SeedSequence`
1114
initialized with the same 256-bits of entropy taken from random.org.
@@ -14,18 +17,52 @@ initialized with the same 256-bits of entropy taken from random.org.
1417

1518
Notes
1619
-----
17-
ꭞ Testing is an on-going process. All generators will be tested to at east 4TB.
18-
19-
¹ Failed at 512GB.
20+
¹ Failures at or before 128GB were generated by tests that used the expanded
21+
set of tests and extra bt folds (``-te 1`` and ``-tf 2``). Failures at sample
22+
sizes above 128GB were produces using the default configuration
23+
(``-te 0`` and ``-tf 0``).
2024

2125
² PCG64DXSM and PCG64(variant=dxsm) are identical and so the latter not separately reported.
2226

2327
³ SFC64(k=weyl) uses distinct Weyl increments that have 50% or fewer non-zero bits.
2428

25-
⁴ The Mersenne Twisters begin to fail at 256GB. This is a known limitation of MT-family
29+
⁴ The Mersenne Twisters begin to fail at 64GB. This is a known limitation of MT-family
2630
generators. These should not be used in large studies except when backward compatibility
2731
is required.
2832

2933
⁵ Identical output to the version included in NumPy 1.19.
3034

3135
.. _PractRand: http://pracrand.sourceforge.net/
36+
37+
Example Configuration
38+
---------------------
39+
All configurations are constructed using the same template. The code below tests a
40+
configuration using 8,196 streams of :class:`~randomgen.aes.AESCounter`. The other
41+
configurations simply make changes to either ``JUMPED`` or ``STREAMS``.
42+
43+
::
44+
45+
import numpy as np
46+
47+
import randomgen as rg
48+
49+
ENTROPY = 86316980830225721106033794313786972513572058861498566720023788662568817403978
50+
JUMPED = False
51+
STREAMS = 8196
52+
BIT_GENERATOR_KWARGS = {}
53+
54+
SEED_SEQ = np.random.SeedSequence(ENTROPY)
55+
56+
57+
BASE_GEN = rg.AESCounter(SEED_SEQ, **BIT_GENERATOR_KWARGS)
58+
if STREAMS == 1:
59+
bit_gens = [BASE_GEN]
60+
elif JUMPED:
61+
bit_gens = [BASE_GEN]
62+
for _ in range(STREAMS - 1):
63+
bit_gens.append(bit_gens[-1].jumped())
64+
else:
65+
bit_gens = []
66+
for child in SEED_SEQ.spawn(STREAMS):
67+
bit_gens.append(rg.AESCounter(child, **BIT_GENERATOR_KWARGS))
68+
output = 64

0 commit comments

Comments
 (0)