How to test a random generator
Four tests, the degrees of freedom for each, our own numbers, and the deliberately broken implementation we run through the same battery to prove the battery detects anything at all.
Updated 2026-09-05
What you are testing, and what you are not
A statistical battery on generated output detects one class of defect: a broken mapping between the entropy source and the characters. Modulo bias, a required character forced into a fixed position, a biased shuffle, a duplicated alphabet. These are structural, they leave a signature in the frequencies, and they are found in minutes.
What it cannot detect is a weak entropy source. Math.floor(Math.random() * n) produces a beautifully uniform distribution. So does a Mersenne Twister, so does an LCG with a decent multiplier, so does any pseudorandom generator worth the name. Uniformity and unpredictability are different properties, and only one of them is visible in the output.
The audit behind this site makes the point concretely. One competitor generating server-side by an undisclosed method returned a chi-square of 55.97 on 61 degrees of freedom over 160,000 characters, which is a comfortable pass and proves nothing about the algorithm. Another, generating client-side with Math.random and a correct multiply-and-floor mapping, returned 38.9 on 35 degrees of freedom over 3.6 million draws. Both distributions are sound. One of them is predictable.
So run the tests, and be precise about the claim they support. They demonstrate that the mapping layer is correct. The entropy source is established by reading the code, not by testing the output.
This is the honest limit of every published randomness result, including ours. A chi-square pass is evidence that the implementation does not corrupt its input. It is not evidence that the input was unpredictable.
Test one: character frequency
Count every character produced across the whole sample and compare the counts against a flat expectation. The statistic is the sum over symbols of the squared deviation divided by the expected count, and it follows a chi-square distribution with N minus 1 degrees of freedom, where N is the alphabet size.
The degrees of freedom matter and are routinely got wrong. For a 62-character alphabet it is 61, not 62, because the counts sum to a known total and the last one is determined by the others. Using 62 shifts the critical value and makes the test slightly too permissive.
Sample size determines what you can detect. To find a bias of a given magnitude you need enough characters that the deviation exceeds the noise, and for the classic byte-modulo bias, which shifts eight symbols by 25 percent, a few hundred thousand characters is far more than enough.
Our own run uses 25,000 strings of 20 characters over a 62-symbol alphabet: 500,000 characters, an expected count of 8,065 per symbol. The statistic was 72.51 on 61 degrees of freedom, p equals 0.149.
Test two: per-position
The frequency test is blind to position. A generator that places a digit at index zero every time and is perfectly uniform everywhere else will pass it comfortably, because the aggregate counts still come out flat.
The fix is to run the same test independently at each index and pool the statistics. With L positions and an alphabet of N, the pooled degrees of freedom are L times N minus 1. For 20 positions over 62 symbols that is 1,220.
This is the test that catches the most common real-world implementation bug, which is satisfying a must-include-a-digit rule by overwriting a fixed slot. It is also the test that catches a shuffle applied only to part of the string.
Report the worst individual position alongside the pooled figure. A pooled pass can hide a single bad index if the other nineteen are clean, so the per-position minimum p-value is the number that actually tells you something. Ours was 0.0521 at index 1, against a pooled statistic of 1,171.06 on 1,220 degrees of freedom, p equals 0.839.
Test three: serial pairs
Frequency tests treat characters as independent draws. Serial correlation is a different failure: each character may be individually uniform while the transition from one to the next is not, which is what a poorly reseeded pool or a stateful mapping produces.
Tabulate every adjacent pair into an N by N grid and run the chi-square over the whole grid. Degrees of freedom are N squared minus 1, which for 62 symbols is 3,843. That is a large table and it needs a correspondingly large sample; with 3,844 cells you want at least a few dozen expected observations per cell for the approximation to hold.
Our run gives 475,000 adjacent pairs, which is 19 per string across 25,000 strings, and an expected count of 123.6 per cell. The statistic was 3,731.71 on 3,843 degrees of freedom, p equals 0.899.
Test four: adjacent repeats
The simplest of the four and the easiest to interpret. Count how often a character equals the one before it. Under uniformity the rate is exactly 1 over N, so for 62 symbols it is 1.6129 percent. Compare the observed count against that with a two-sided z test.
It is worth running because it is the test whose result you can explain to anyone, and because a startlingly common intuition is that a good generator should avoid repeats. It should not: a repeat every 62 characters is exactly what randomness looks like, and a generator that suppresses them is the one with the bug.
Ours observed 7,704 repeats against 7,661 expected, a z of 0.492, p equals 0.623.
The control, which is the entire point
A test suite that has only ever passed has never been calibrated. Before any of the numbers above mean anything, you have to demonstrate that the battery fails when it should.
So we run a deliberately biased implementation through the identical battery at the identical sample size. The control is the naive byte modulo mapping: draw a uint8, take it modulo 62, index the alphabet. Because 256 is 4 times 62 plus 8, eight symbols get five preimages and the other 54 get four, so the most likely symbol is exactly 1.25 times more likely than the least.
The battery catches it, decisively, on three tests out of four. Character frequency returns 3,118.05 on 61 degrees of freedom. Per-position returns 4,253.66 on 1,220, with a worst single position p-value of 6.50e-37. Serial pairs returns 9,634.82 on 3,843. All three p-values round to zero at double precision.
The suite also carries self-checks on its own statistics: the chi-square tail function is verified against known critical values at several degrees of freedom, and the normal tail against known z quantiles, so a wrong p-value cannot silently produce a pass.
| Test | df | Ours | p | Control | p | Caught? |
|---|---|---|---|---|---|---|
| Character frequency | 61 | 72.509 | 0.149 | 3118.048 | 0 | Yes |
| Positional bias | 1220 | 1171.058 | 0.839 | 4253.658 | 0 | Yes |
| Serial pairs | 3843 | 3731.711 | 0.899 | 9634.817 | 0 | Yes |
| Adjacent repeats | z test | z = 0.492 | 0.623 | z = 1.678 | 0.093 | No |
The test that missed it, and why
The adjacent-repeat test passed the biased control. That is not a defect in the run, it is a property of the test, and reporting it is more useful than quietly dropping the row.
The repeat rate under a non-uniform marginal distribution is the sum of the squared symbol probabilities. Under the byte-modulo bias, eight symbols sit at 5/256 and 54 sit at 4/256, so that sum is 0.016235 against the uniform 0.016129. The bias raises the repeat rate by 0.66 percent in relative terms.
Over 475,000 pairs that is about 51 extra repeats. The standard deviation of the count under the null is roughly 87. A shift of 51 against a noise floor of 87 is invisible, and no sample size you would realistically run changes that by much, because the effect enters at second order in the deviation while the noise falls only as the square root of the sample.
The general lesson: the adjacent-repeat test is a test of serial independence, not of marginal uniformity. It will find a generator that avoids or favours repeats. It will not find a skew in how often each symbol appears, which is exactly what modulo bias is. Run all four, and know which one is answering which question.
Running this on something you did not write
For a client-side generator, collect a sample by driving the page and reading the output, then run the same four tests. The two things to fix before you start are sample size and alphabet: you need the exact alphabet the tool uses, including whether it deduplicates, or your expected counts are wrong and every result is meaningless.
For a server-side generator, the sample is expensive. Rate limits are common, several of the audited tools cap requests at 60 per minute, and the arithmetic on how long a 500,000-character sample takes at that rate is usually the end of the exercise. This is a real asymmetry: local generation can be tested exhaustively by anyone, and remote generation mostly cannot.
Whatever you measure, publish the sample size, the alphabet, the degrees of freedom, the statistic and the significance level. A verdict without those is not a result. And publish the control run alongside it, because a battery that has never failed is not evidence of anything.
Doing this in your own code
Node.js, the whole battery
// Character frequency: chi-square goodness of fit, df = N - 1.
function chiSquareFrequency(strings, alphabet) {
const N = alphabet.length;
const idx = new Map([...alphabet].map((c, i) => [c, i]));
const obs = new Array(N).fill(0);
let total = 0;
for (const s of strings) {
for (const c of s) { obs[idx.get(c)]++; total++; }
}
const exp = total / N;
let x2 = 0;
for (const o of obs) x2 += ((o - exp) ** 2) / exp;
return { statistic: x2, df: N - 1, expected: exp };
}
// Per position: the same test run independently at each index, pooled.
// df = L * (N - 1). This is the one that catches a required character
// forced into a fixed slot.
function chiSquarePositional(strings, alphabet, length) {
const N = alphabet.length;
const idx = new Map([...alphabet].map((c, i) => [c, i]));
let x2 = 0;
for (let p = 0; p < length; p++) {
const obs = new Array(N).fill(0);
for (const s of strings) obs[idx.get(s[p])]++;
const exp = strings.length / N;
for (const o of obs) x2 += ((o - exp) ** 2) / exp;
}
return { statistic: x2, df: length * (N - 1) };
}
// Adjacent repeats: a two-sided z test against p = 1/N.
function adjacentRepeats(strings, N) {
let repeats = 0, pairs = 0;
for (const s of strings) {
for (let i = 1; i < s.length; i++) { if (s[i] === s[i - 1]) repeats++; pairs++; }
}
const p = 1 / N;
const exp = pairs * p;
const sd = Math.sqrt(pairs * p * (1 - p));
return { observed: repeats, expected: exp, z: (repeats - exp) / sd };
}Run every one of these against a deliberately broken generator as well. A battery that has never failed has never been calibrated.
The control you must also run
// The naive mapping, which is what you are trying to detect.
// 256 = 4 * 62 + 8, so the first 8 symbols get 5 preimages and the
// other 54 get 4. The ratio is exactly 1.25.
function biasedString(length, alphabet) {
const bytes = crypto.getRandomValues(new Uint8Array(length));
let out = "";
for (const b of bytes) out += alphabet[b % alphabet.length];
return out;
}
// Feed it through the identical battery. If it passes, your tests are
// not sensitive enough at your sample size, and a pass on your real
// generator means nothing.Questions
Can a chi-square test prove a generator is cryptographically secure?
No, and this is the most important limitation to be clear about. Math.random passes a chi-square test easily, as does any decent pseudorandom generator. The test proves the mapping from bits to characters preserves uniformity. Whether the bits were unpredictable is a question about the source, answered by reading the code and identifying which system call it reaches.
What significance level should I use?
We use 0.001, which is deliberately strict. At 0.05 you expect a false failure once every twenty tests, and with four tests run across several configurations that produces alarming-looking results on a correct generator often enough to be a nuisance. A real bias produces p-values that round to zero, so a tight threshold costs nothing in sensitivity to the defects that matter.
How large a sample do I need?
For the frequency and repeat tests, a few hundred thousand characters is comfortable. The binding constraint is the serial-pair test, which has N squared cells: at 62 symbols that is 3,844 cells, and you want enough expected observations in each for the chi-square approximation to be valid. Our run gives 123.6 per cell from 500,000 characters, which is sound. At a 94-character alphabet the same test needs about 2.4 times the sample.
Why do you publish a failing result?
Because the control is supposed to fail, and because the adjacent-repeat row shows it passing a generator we know is broken. Both facts are more informative than a table of four passes. A suite that reports only successes gives a reader no way to judge whether it can detect anything.
Do these tests replace something like Dieharder or TestU01?
No. Those batteries test raw bit streams from an entropy source and are far more thorough at that job. These four test the layer above: whether a correct source has been correctly mapped onto an alphabet. Different question, different tool. If you are writing a generator rather than consuming one, run both.