For completion, the remaining |Y-p| cases for determining with 95% certainty that the underlying theoretical probability is not 50% (ie. p != 0.5). This was done by laboriously going through cumulative binomial distribution calculations, where the normal approximation is no longer valid.
|Y-p| = 0.20 --> n > 29
|Y-p| = 0.25 --> n > 19
|Y-p| = 0.30 --> n > 13
|Y-p| = 0.35 --> n > 10
|Y-p| = 0.40 --> n > 8
|Y-p| = 0.45 --> n > 7
Within a generic four-five hour DnD game with a few dozen or so d20 rolls made, one would certainly notice a bonus/penalty of 5 or greater to the d20 rolls.
|Y-p| = 0.20 --> n > 29
|Y-p| = 0.25 --> n > 19
|Y-p| = 0.30 --> n > 13
|Y-p| = 0.35 --> n > 10
|Y-p| = 0.40 --> n > 8
|Y-p| = 0.45 --> n > 7
Within a generic four-five hour DnD game with a few dozen or so d20 rolls made, one would certainly notice a bonus/penalty of 5 or greater to the d20 rolls.