In-game probability in practice - part 4.

For completion, the remaining |Y-p| cases for determining with 95% certainty that the underlying theoretical probability is not 50% (ie. p != 0.5). This was done by laboriously going through cumulative binomial distribution calculations, where the normal approximation is no longer valid.

|Y-p| = 0.20 --> n > 29
|Y-p| = 0.25 --> n > 19
|Y-p| = 0.30 --> n > 13
|Y-p| = 0.35 --> n > 10
|Y-p| = 0.40 --> n > 8
|Y-p| = 0.45 --> n > 7

Within a generic four-five hour DnD game with a few dozen or so d20 rolls made, one would certainly notice a bonus/penalty of 5 or greater to the d20 rolls.

In-game probability in practice - part 3.

Let's examine the scenario where one keeps on getting a practical in-game probability Y (from rolling a d20 die "n" times) which appears to consistently deviate from a presumed theoretical probability "p".

We will assume the underlying distribution for rolling a die "n" times is binomial with a theoretical probability p. For n large enough (ie. np(1-p)>10), the normal approximation for the binomial distribution can be used.

The test statistic "z" used to test whether the underlying distribution indeed has a theoretical probability of p, will follow a standard normal distribution with:

z = (Y-p)/sqrt[p(1-p)/n]

For a 95% certainty that the theoretical probability is something other than p, one requires |z| > 1.96 (from a table of areas under the normal distribution). Similarly for a 99% certainty, |z| > 2.575.


For some concrete numbers, we'll look at the case where the presumed theoretical probability is 50% (p = 0.5). From this, we can calculate what the minimum number "n" of d20 die rolls it takes to determine with 95% certainty that the underlying theoretical probability is not 50% (p != 0.5).

Doing some algebra, we get for "n":

n > [1.96/(Y-p)]^2 *[p(1-p)]

for 95% certainty that the theoretical probability is not p. (Similarly for the case of 99% certainty that the theoretical probability is not p, we get n > [2.575/(Y-p)]^2 *[p(1-p)] ).


For the case where |Y-p| = 0.05 which corresponds to a constant +1 bonus or -1 penalty to a d20 roll, one needs to do n > 384 rolls of a d20 in order to determine with 95% certainty that the underlying theoretical probability is not 50% (ie. p != 0.5).

Similarly for different |Y-p| values:

|Y-p| = 0.05 --> n > 384
|Y-p| = 0.10 --> n > 96
|Y-p| = 0.15 --> n > 43
(For |Y-p| = 0.20, the normal approximation to binomial is no longer valid).


For the case where one wants 99% certainty that the underlying theoretical probability is not 50%, we get for different |Y-p| values:

|Y-p| = 0.05 --> n > 663
|Y-p| = 0.10 --> n > 166
|Y-p| = 0.15 --> n > 74
|Y-p| = 0.20 --> n > 41
(For |Y-p| = 0.25, the normal approximation to binomial is no longer valid).

These results suggest that in a generic four-five hour DnD session, there may not be enough d20 die rolls (of a few dozen) with bonuses/penalties of 1 or 2, to determine whether the underlying theoretical probability "p" is not 50% (p != 0.5) with a 95% (or 99%) certainty. One needs more than a hundred or so d20 die rolls to make this determination.

When one is dealing with bonuses/penalties of 4 (or greater), there may be enough d20 die rolls (of a few dozen) in a generic four-five hour DnD game session, such that one can determine with 95% (or 99%) certainty that the underlying theoretical probability "p" is not 50% (p != 0.5).

Bonuses/penalties of 3, are the borderline cases where there may be enough d20 die rolls to determine if the underlying theoretical probability "p" is not 50% (p != 0.5) with 95% or 99% certainty. (One requires at least 43 d20 die rolls to determine this with 95% certainty).

A +5 or +6 magic weapon being used at heroic tier in 4E DnD, or the players are subjected to an "aura" which causes a -4 or -5 penalty to hit, would be noticeable and conclude with 95% (or better) certainty in a four/five hour DnD session that the underlying theoretical probability "p" is not 50% (p != 0.5).


Possibly this explains why so many bonuses/penalties in 4E DnD are +/-1 or +/-2 to the d20 roll, along with all kinds of kludges to prevent the stacking of too many bonuses/penalties on top of one another. Essentially they're attempting to maintain an illusion of "always fighting orcs" with the underlying theoretical probability "p" appearing to be 50% (p=0.5) over a four-five hour DnD game session. (One would need over a hundred or so d20 die rolls with bonuses/penalties of 2, in order to determine with a 95% or better certainty, that the underlying theoretical probability "p" is not 50%).

In-game probability in practice - part 2.

Let's examine the practical in-game probability "Y" in a typical 4-5 hour DnD game. ("Y" is the number of successful die rolls divided by the total number of die rolls including the misses/failures).

Recall the 95% confidence interval for the in-game probability "Y" (in the normal approximation to binomial distribution):

p - d < Y < p + d

where d = 1.96 * sqrt[p(1-p)/n] and p = theoretical probability.

(For a 99% confidence interval, d = 2.575 * sqrt[p(1-p)/n]).

For this normal approximation to be viable, np(1-p) > 10.


In a typical DnD game, the theoretical probability of success "p" for a d20 roll whether for attacks, skill checks, etc ... is around 50%. (For this normal approximation to binomial distribution to be valid, one needs n > 40 for p=0.5).

If in a typical DnD game a player is doing several dozen or so d20 rolls in the session, let's look at the deviation "d" when n = 41 (ie. borderline case).

So for n = 41 rolls and theoretical probability of success p = 0.5, the deviation "d" of the in-game probability "Y" from "p" for a 95% confidence interval is:

d = 1.96* sqrt[p(1-p)/n] = 0.15

This means that when one is doing 41 rolls of a d20 through a generic 4-5 hour DnD game session, there is a 95% probability that the practical in-game probability "Y" is within the interval 0.35 < Y < 0.65. So 95% of the time, the in-game probability "Y" sees when one is rolling a d20 41 times in a game, can vary from 35% to 65% for a theoretical probability of p=50% (ie. rolling greater than or equal to an 11 for a success on a d20).


If we do this calculation for a 99% confidence interval, the deviation "d" of the in-game probability "Y" from "p" for a 99% confidence interval is:

d = 2.575* sqrt[p(1-p)/n] = 0.20

This means that when one is doing 41 rolls of a d20 through a generic 4-5 hour DnD game, there is a 99% probability that the practical in-game probability "Y" is within the interval 0.30 < Y < 0.70. So 99% of the time, the in-game probability "Y" sees when one is rolling a d20 41 times in a game, can vary from 30% to 70% for a theoretical probability of p=50% (ie. rolling greater than or equal to an 11 for a success on a d20).

In-game probability in practice - part 1.

As one may (or may not) have noticed, rolling a die multiple times does not always follow the theoretical probabilities that closely. For example, it's possible to roll a lot of low numbers on a d20 in a streak. It's also possible to roll a lot of high numbers on a d20 in a row (ie. a hot hand).

So how does the relative frequency of successful die rolls, relate to the theoretical probability for rolling a success?

It turns out, the act of rolling a die multiple times follows a binomial distribution.


To introduce some nomenclature, let's use:

X = number of successes
n = number of trials
p = theoretical probability of success
1-p = theoretical probability of failure

In the case of rolling a die "n" number of times, one will have "X" number of success. (For example, rolling a d20).

The theoretical probability "p" is what one would expect, from examining the DC's for a success. For example, rolling an 11 or over on a d20 is a 50% theoretical probability.

For a binomial distribution, the average value is m = np and the standard deviation is stdev = sqrt[np(1-p)]. For a large enough number of trials "n", one can approximate the binomial distribution with a normal distribution with average value m = np and standard deviation stdev = sqrt[np(1-p)].

(From a probability textbook, such as Sheldon Ross' textbooks on probability, the criteria for the normal approximation being reliable is np(1-p) > 10).

So for rolling a die "n" times and getting "X" successes (where "n" is large), it will follow a standard normal distribution (ie. with mean 0 and variance 1), with:

Z = (X - np)/sqrt[np(1-p)]

Doing some algebra, Z = [(X/n) - p]/sqrt[p(1-p)/n]. In this form, let Y = X/n represents the relative frequency of successes of rolling a die. This would be an estimate "Y" of the theoretical probability "p", from rolling a die "n" times and getting "X" number of successes. We would like to know how "Y" is related to the theoretical "p".

Since Z = (Y-p)/sqrt[p(1-p)/n] approximately follows a standard normal distribution, this means the confidence intervals can be determined for a range of Z's.

For a 95% confidence interval, Z will fall between -1.96 and 1.96 (from a table of areas of a normal distribution). (For a 99% confidence interval, Z will fall between -2.575 an 2.575).

In the case of a 95% confidence interval |Z| < 1.96, this means:

Probability ( |Y-p| < 1.96 * sqrt[p(1-p)/n] ) = 0.95

(Similarly for a 99% confidence interval, Probability ( |Y-p| < 2.575 * sqrt[p(1-p)/n] ) = 0.99 ).

Let the deviation d = 1.96 * sqrt[p(1-p)/n], which represents the deviation of the estimate "Y" from the theoretical probability "p".

From this we can determine how many number of trials "n" (ie. rolls of a die) it takes, such that the deviation between the estimate "Y" and theoretical probability "p" will stay within "d" with a 95% probability.

Doing some algebra, we get "n" in terms of "d":

n = (1.96)^2 * p(1-p)/(d^2)

For various theoretical probabilities "p" and deviations "d", we get:

(For a d20, a 5% deviation d = 0.05 starts to overlap between different values on the d20. If one wants more precision, one can use a 1% deviation d = 0.01).

d = 0.05
p = 0.50 ---> n = 384
p = 0.40 ---> n = 369
p = 0.30 ---> n = 323
p = 0.20 ---> n = 246
p = 0.10 ---> n = 138

d = 0.01
p = 0.50 ---> n = 9604
p = 0.40 ---> n = 9220
p = 0.30 ---> n = 8067
p = 0.20 ---> n = 6147
p = 0.10 ---> n = 3457

These results mean that for 95% of the time when rolling a die to hit a particular DC, the practical in-game probability (ie. number of successes divided by the total number of die rolls including misses/failures) deviating less than 5% from the "theoretical probability", would require at least a few hundred die rolls. (For a 1% deviation between the practical in-game and theoretical probabilities, it would require several thousand die rolls).

Not too surprising that in a generic four-five hour D&D game session where one only does a few dozen or so d20 rolls, the practical in-game probability "Y" does not always converge to the theoretical probability "p". One needs at least several hundred d20 rolls to see this convergence within a 5% tolerance band, 95% of the time.

One shouldn't be shocked at seeing somebody with a "hot hand" streak of rolling lots of high numbers on a d20, or somebody else on a losing streak of rolling lots of low numbers on a d20.

In-game probability in practice.

We'll be reposting the "In-game probability in practice" posts from therpgsite.