Let's examine the scenario where one keeps on getting a practical in-game probability Y (from rolling a d20 die "n" times) which appears to consistently deviate from a presumed theoretical probability "p".

We will assume the underlying distribution for rolling a die "n" times is binomial with a theoretical probability p. For n large enough (ie. np(1-p)>10), the normal approximation for the binomial distribution can be used.

The test statistic "z" used to test whether the underlying distribution indeed has a theoretical probability of p, will follow a standard normal distribution with:

z = (Y-p)/sqrt[p(1-p)/n]

For a 95% certainty that the theoretical probability is something other than p, one requires |z| > 1.96 (from a table of areas under the normal distribution). Similarly for a 99% certainty, |z| > 2.575.

For some concrete numbers, we'll look at the case where the presumed theoretical probability is 50% (p = 0.5). From this, we can calculate what the minimum number "n" of d20 die rolls it takes to determine with 95% certainty that the underlying theoretical probability is not 50% (p != 0.5).

Doing some algebra, we get for "n":

n > [1.96/(Y-p)]^2 *[p(1-p)]

for 95% certainty that the theoretical probability is not p. (Similarly for the case of 99% certainty that the theoretical probability is not p, we get n > [2.575/(Y-p)]^2 *[p(1-p)] ).

For the case where |Y-p| = 0.05 which corresponds to a constant +1 bonus or -1 penalty to a d20 roll, one needs to do n > 384 rolls of a d20 in order to determine with 95% certainty that the underlying theoretical probability is not 50% (ie. p != 0.5).

Similarly for different |Y-p| values:

|Y-p| = 0.05 --> n > 384

|Y-p| = 0.10 --> n > 96

|Y-p| = 0.15 --> n > 43

(For |Y-p| = 0.20, the normal approximation to binomial is no longer valid).

For the case where one wants 99% certainty that the underlying theoretical probability is not 50%, we get for different |Y-p| values:

|Y-p| = 0.05 --> n > 663

|Y-p| = 0.10 --> n > 166

|Y-p| = 0.15 --> n > 74

|Y-p| = 0.20 --> n > 41

(For |Y-p| = 0.25, the normal approximation to binomial is no longer valid).

These results suggest that in a generic four-five hour DnD session, there may not be enough d20 die rolls (of a few dozen) with bonuses/penalties of 1 or 2, to determine whether the underlying theoretical probability "p" is not 50% (p != 0.5) with a 95% (or 99%) certainty. One needs more than a hundred or so d20 die rolls to make this determination.

When one is dealing with bonuses/penalties of 4 (or greater), there may be enough d20 die rolls (of a few dozen) in a generic four-five hour DnD game session, such that one can determine with 95% (or 99%) certainty that the underlying theoretical probability "p" is not 50% (p != 0.5).

Bonuses/penalties of 3, are the borderline cases where there may be enough d20 die rolls to determine if the underlying theoretical probability "p" is not 50% (p != 0.5) with 95% or 99% certainty. (One requires at least 43 d20 die rolls to determine this with 95% certainty).

A +5 or +6 magic weapon being used at heroic tier in 4E DnD, or the players are subjected to an "aura" which causes a -4 or -5 penalty to hit, would be noticeable and conclude with 95% (or better) certainty in a four/five hour DnD session that the underlying theoretical probability "p" is not 50% (p != 0.5).

Possibly this explains why so many bonuses/penalties in 4E DnD are +/-1 or +/-2 to the d20 roll, along with all kinds of kludges to prevent the stacking of too many bonuses/penalties on top of one another. Essentially they're attempting to maintain an illusion of "always fighting orcs" with the underlying theoretical probability "p" appearing to be 50% (p=0.5) over a four-five hour DnD game session. (One would need over a hundred or so d20 die rolls with bonuses/penalties of 2, in order to determine with a 95% or better certainty, that the underlying theoretical probability "p" is not 50%).