In-game probability in practice - part 5.

Here are the 95% and 99% confidence intervals for the practical in-game probabilities "Y" after rolling a die n times, with an underlying theoretical probability of 50% (p = 0.5).

(They were calculated using a program which implemented the Clopper-Pearson method, when the normal approximation is no longer valid for n < 40).

95%

n = 2 --> (0.0126, 0.9874)
n = 4 --> (0.0676, 0.9324)
n = 6 --> (0.1181, 0.8819)
n = 8 --> (0.1507, 0.8430)
n = 10 --> (0.1871, 0.8129)
n = 12 --> (0.2109, 0.7891)
n = 14 --> (0.2304, 0.7696)
n = 16 --> (0.2465, 0.7535)
n = 18 --> (0.2602, 0.7381)
n = 20 --> (0.2720, 0.7280)
n = 22 --> (0.2822, 0.7178)
n = 24 --> (0.2912, 0.7088)
n = 26 --> (0.2993, 0.7007)
n = 28 --> (0.3065, 0.6935)
n = 30 --> (0.3130, 0.6870)
n = 32 --> (0.3189, 0.6811)
n = 34 --> (0.3243, 0.6757)
n = 36 --> (0.3292, 0.6708)
n = 38 --> (0.3338, 0.6662)
n = 40 --> (0.3380, 0.6620)

99%

n = 2 --> (0.0025, 0.9975)
n = 4 --> (0.0294, 0.9706)
n = 6 --> (0.0663, 0.9337)
n = 8 --> (0.0999, 0.9001)
n = 10 --> (0.1283, 0.8717)
n = 12 --> (0.1522, 0.8478)
n = 14 --> (0.1724, 0.8276)
n = 16 --> (0.1897, 0.8103)
n = 18 --> (0.2047, 0.7953)
n = 20 --> (0.2177, 0.7823)
n = 22 --> (0.2293, 0.7707)
n = 24 --> (0.2396, 0.7604)
n = 26 --> (0.2489, 0.7511)
n = 28 --> (0.2572, 0.7428)
n = 30 --> (0.2649, 0.7351)
n = 32 --> (0.2718, 0.7282)
n = 34 --> (0.2782, 0.7281)
n = 36 --> (0.2841, 0.7152)
n = 38 --> (0.2895, 0.7105)
n = 40 --> (0.2946, 0.7054)


As one can see for less than 4 or 5 rolls of a d20, with a 95% to 99% certainty, the practical in-game probabilities "Y" include the cases of 5% (ie. only hitting on a d20 roll of 20) and 95% (ie. hit on any roll of a d20, except on a 1).

Also in a generic DnD combat encounter which lasts 8 rounds or less, with 95% to 99% certainty, it should not be surprising at all to see a player which only hits once on all of their d20 rolls during the encounter (ie. Pinky's cousin), or a player which only misses once on all their d20 rolls during the encounter (ie. hot hand).

(Pinky is a "cursed" d20 which never rolls a success).

For completion, the number of rounds "n" where it is not surprising to see one with "Pinky's cousin" or a "hot hand", with 95% to 99% certainty.

One hit only (or only one miss on a "hot hand")

95% certainty --> n < 8
99% certainty --> n < 9

Two hits only (or only two misses on a "hot hand")

95% certainty --> n < 11
99% certainty --> n < 13

Three hits only (or only three misses on a "hot hand")

95% certainty --> n < 14
99% certainty --> n < 16

Four hits only (or only four misses on a "hot hand")

95% certainty --> n < 17
99% certainty --> n < 19

Five hits only (or only five misses on a "hot hand")

95% certainty --> n < 19
99% certainty --> n < 22

---

In-game probability in practice - part 4.

For completion, the remaining |Y-p| cases for determining with 95% certainty that the underlying theoretical probability is not 50% (ie. p != 0.5). This was done by laboriously going through cumulative binomial distribution calculations, where the normal approximation is no longer valid.

|Y-p| = 0.20 --> n > 29
|Y-p| = 0.25 --> n > 19
|Y-p| = 0.30 --> n > 13
|Y-p| = 0.35 --> n > 10
|Y-p| = 0.40 --> n > 8
|Y-p| = 0.45 --> n > 7

Within a generic four-five hour DnD game with a few dozen or so d20 rolls made, one would certainly notice a bonus/penalty of 5 or greater to the d20 rolls.

In-game probability in practice - part 3.

Let's examine the scenario where one keeps on getting a practical in-game probability Y (from rolling a d20 die "n" times) which appears to consistently deviate from a presumed theoretical probability "p".

We will assume the underlying distribution for rolling a die "n" times is binomial with a theoretical probability p. For n large enough (ie. np(1-p)>10), the normal approximation for the binomial distribution can be used.

The test statistic "z" used to test whether the underlying distribution indeed has a theoretical probability of p, will follow a standard normal distribution with:

z = (Y-p)/sqrt[p(1-p)/n]

For a 95% certainty that the theoretical probability is something other than p, one requires |z| > 1.96 (from a table of areas under the normal distribution). Similarly for a 99% certainty, |z| > 2.575.


For some concrete numbers, we'll look at the case where the presumed theoretical probability is 50% (p = 0.5). From this, we can calculate what the minimum number "n" of d20 die rolls it takes to determine with 95% certainty that the underlying theoretical probability is not 50% (p != 0.5).

Doing some algebra, we get for "n":

n > [1.96/(Y-p)]^2 *[p(1-p)]

for 95% certainty that the theoretical probability is not p. (Similarly for the case of 99% certainty that the theoretical probability is not p, we get n > [2.575/(Y-p)]^2 *[p(1-p)] ).


For the case where |Y-p| = 0.05 which corresponds to a constant +1 bonus or -1 penalty to a d20 roll, one needs to do n > 384 rolls of a d20 in order to determine with 95% certainty that the underlying theoretical probability is not 50% (ie. p != 0.5).

Similarly for different |Y-p| values:

|Y-p| = 0.05 --> n > 384
|Y-p| = 0.10 --> n > 96
|Y-p| = 0.15 --> n > 43
(For |Y-p| = 0.20, the normal approximation to binomial is no longer valid).


For the case where one wants 99% certainty that the underlying theoretical probability is not 50%, we get for different |Y-p| values:

|Y-p| = 0.05 --> n > 663
|Y-p| = 0.10 --> n > 166
|Y-p| = 0.15 --> n > 74
|Y-p| = 0.20 --> n > 41
(For |Y-p| = 0.25, the normal approximation to binomial is no longer valid).

These results suggest that in a generic four-five hour DnD session, there may not be enough d20 die rolls (of a few dozen) with bonuses/penalties of 1 or 2, to determine whether the underlying theoretical probability "p" is not 50% (p != 0.5) with a 95% (or 99%) certainty. One needs more than a hundred or so d20 die rolls to make this determination.

When one is dealing with bonuses/penalties of 4 (or greater), there may be enough d20 die rolls (of a few dozen) in a generic four-five hour DnD game session, such that one can determine with 95% (or 99%) certainty that the underlying theoretical probability "p" is not 50% (p != 0.5).

Bonuses/penalties of 3, are the borderline cases where there may be enough d20 die rolls to determine if the underlying theoretical probability "p" is not 50% (p != 0.5) with 95% or 99% certainty. (One requires at least 43 d20 die rolls to determine this with 95% certainty).

A +5 or +6 magic weapon being used at heroic tier in 4E DnD, or the players are subjected to an "aura" which causes a -4 or -5 penalty to hit, would be noticeable and conclude with 95% (or better) certainty in a four/five hour DnD session that the underlying theoretical probability "p" is not 50% (p != 0.5).


Possibly this explains why so many bonuses/penalties in 4E DnD are +/-1 or +/-2 to the d20 roll, along with all kinds of kludges to prevent the stacking of too many bonuses/penalties on top of one another. Essentially they're attempting to maintain an illusion of "always fighting orcs" with the underlying theoretical probability "p" appearing to be 50% (p=0.5) over a four-five hour DnD game session. (One would need over a hundred or so d20 die rolls with bonuses/penalties of 2, in order to determine with a 95% or better certainty, that the underlying theoretical probability "p" is not 50%).

In-game probability in practice - part 2.

Let's examine the practical in-game probability "Y" in a typical 4-5 hour DnD game. ("Y" is the number of successful die rolls divided by the total number of die rolls including the misses/failures).

Recall the 95% confidence interval for the in-game probability "Y" (in the normal approximation to binomial distribution):

p - d < Y < p + d

where d = 1.96 * sqrt[p(1-p)/n] and p = theoretical probability.

(For a 99% confidence interval, d = 2.575 * sqrt[p(1-p)/n]).

For this normal approximation to be viable, np(1-p) > 10.


In a typical DnD game, the theoretical probability of success "p" for a d20 roll whether for attacks, skill checks, etc ... is around 50%. (For this normal approximation to binomial distribution to be valid, one needs n > 40 for p=0.5).

If in a typical DnD game a player is doing several dozen or so d20 rolls in the session, let's look at the deviation "d" when n = 41 (ie. borderline case).

So for n = 41 rolls and theoretical probability of success p = 0.5, the deviation "d" of the in-game probability "Y" from "p" for a 95% confidence interval is:

d = 1.96* sqrt[p(1-p)/n] = 0.15

This means that when one is doing 41 rolls of a d20 through a generic 4-5 hour DnD game session, there is a 95% probability that the practical in-game probability "Y" is within the interval 0.35 < Y < 0.65. So 95% of the time, the in-game probability "Y" sees when one is rolling a d20 41 times in a game, can vary from 35% to 65% for a theoretical probability of p=50% (ie. rolling greater than or equal to an 11 for a success on a d20).


If we do this calculation for a 99% confidence interval, the deviation "d" of the in-game probability "Y" from "p" for a 99% confidence interval is:

d = 2.575* sqrt[p(1-p)/n] = 0.20

This means that when one is doing 41 rolls of a d20 through a generic 4-5 hour DnD game, there is a 99% probability that the practical in-game probability "Y" is within the interval 0.30 < Y < 0.70. So 99% of the time, the in-game probability "Y" sees when one is rolling a d20 41 times in a game, can vary from 30% to 70% for a theoretical probability of p=50% (ie. rolling greater than or equal to an 11 for a success on a d20).

In-game probability in practice - part 1.

As one may (or may not) have noticed, rolling a die multiple times does not always follow the theoretical probabilities that closely. For example, it's possible to roll a lot of low numbers on a d20 in a streak. It's also possible to roll a lot of high numbers on a d20 in a row (ie. a hot hand).

So how does the relative frequency of successful die rolls, relate to the theoretical probability for rolling a success?

It turns out, the act of rolling a die multiple times follows a binomial distribution.


To introduce some nomenclature, let's use:

X = number of successes
n = number of trials
p = theoretical probability of success
1-p = theoretical probability of failure

In the case of rolling a die "n" number of times, one will have "X" number of success. (For example, rolling a d20).

The theoretical probability "p" is what one would expect, from examining the DC's for a success. For example, rolling an 11 or over on a d20 is a 50% theoretical probability.

For a binomial distribution, the average value is m = np and the standard deviation is stdev = sqrt[np(1-p)]. For a large enough number of trials "n", one can approximate the binomial distribution with a normal distribution with average value m = np and standard deviation stdev = sqrt[np(1-p)].

(From a probability textbook, such as Sheldon Ross' textbooks on probability, the criteria for the normal approximation being reliable is np(1-p) > 10).

So for rolling a die "n" times and getting "X" successes (where "n" is large), it will follow a standard normal distribution (ie. with mean 0 and variance 1), with:

Z = (X - np)/sqrt[np(1-p)]

Doing some algebra, Z = [(X/n) - p]/sqrt[p(1-p)/n]. In this form, let Y = X/n represents the relative frequency of successes of rolling a die. This would be an estimate "Y" of the theoretical probability "p", from rolling a die "n" times and getting "X" number of successes. We would like to know how "Y" is related to the theoretical "p".

Since Z = (Y-p)/sqrt[p(1-p)/n] approximately follows a standard normal distribution, this means the confidence intervals can be determined for a range of Z's.

For a 95% confidence interval, Z will fall between -1.96 and 1.96 (from a table of areas of a normal distribution). (For a 99% confidence interval, Z will fall between -2.575 an 2.575).

In the case of a 95% confidence interval |Z| < 1.96, this means:

Probability ( |Y-p| < 1.96 * sqrt[p(1-p)/n] ) = 0.95

(Similarly for a 99% confidence interval, Probability ( |Y-p| < 2.575 * sqrt[p(1-p)/n] ) = 0.99 ).

Let the deviation d = 1.96 * sqrt[p(1-p)/n], which represents the deviation of the estimate "Y" from the theoretical probability "p".

From this we can determine how many number of trials "n" (ie. rolls of a die) it takes, such that the deviation between the estimate "Y" and theoretical probability "p" will stay within "d" with a 95% probability.

Doing some algebra, we get "n" in terms of "d":

n = (1.96)^2 * p(1-p)/(d^2)

For various theoretical probabilities "p" and deviations "d", we get:

(For a d20, a 5% deviation d = 0.05 starts to overlap between different values on the d20. If one wants more precision, one can use a 1% deviation d = 0.01).

d = 0.05
p = 0.50 ---> n = 384
p = 0.40 ---> n = 369
p = 0.30 ---> n = 323
p = 0.20 ---> n = 246
p = 0.10 ---> n = 138

d = 0.01
p = 0.50 ---> n = 9604
p = 0.40 ---> n = 9220
p = 0.30 ---> n = 8067
p = 0.20 ---> n = 6147
p = 0.10 ---> n = 3457

These results mean that for 95% of the time when rolling a die to hit a particular DC, the practical in-game probability (ie. number of successes divided by the total number of die rolls including misses/failures) deviating less than 5% from the "theoretical probability", would require at least a few hundred die rolls. (For a 1% deviation between the practical in-game and theoretical probabilities, it would require several thousand die rolls).

Not too surprising that in a generic four-five hour D&D game session where one only does a few dozen or so d20 rolls, the practical in-game probability "Y" does not always converge to the theoretical probability "p". One needs at least several hundred d20 rolls to see this convergence within a 5% tolerance band, 95% of the time.

One shouldn't be shocked at seeing somebody with a "hot hand" streak of rolling lots of high numbers on a d20, or somebody else on a losing streak of rolling lots of low numbers on a d20.

In-game probability in practice.

We'll be reposting the "In-game probability in practice" posts from therpgsite.

4E powers with half-damage on miss.

Let's examine the case for new encounter powers which produce half-damage on a miss. (These are purportedly being introduced in 4E Essentials). Let's assume the new encounter powers still follow the same damage of the Heinsoo 4E D&D encounter powers, but with the "half-damage on miss" tacked on.

For non-striker encounter powers with half-damage on a miss and a probability p of a hitting a monster of the same level, the average damage per round scales approximately as (after level 10):

0.5(p+1)*[level*(average[W])/7 + 7*(level-10)/20]

So as the level goes to infinity, "R" approaches:

R -> 2*ROLE/{(p+1)*[(average[W])/7 + 7/20]}


For different [W] weapons attacking a skirmisher monster (ROLE=8) with the player having a p=50% of hitting the monster, we have average number of rounds "R" as the level goes to infinity:

average[d12] = 6.5 --> R = 8.34
average[d10] = 5.5 --> R = 9.39
average[d8] = 4.5 --> R = 10.74
average[d6] = 3.5 --> R = 12.55
average[d4] = 2.5 --> R = 15.08

On average, for a player hypothetically repeatedly spamming a new encounter power (with half-damage on a miss) against a skirmisher monster of the same level, the average number of rounds "R" to kill the monster is approximately shorter by 45% compared to at-will powers, as the level goes to infinity.


For completeness, there's at-will powers with half-damage on a miss. (IIRC, there's a few monsters with such an at-will power from the 4E MM2 and/or MM3).

With half-damage on a miss, the at-will with half-damage on a miss reduces the number of rounds R to kill a monster by a factor = p/(1+p) in comparison to ordinary at-will powers. (p is the probability of hitting a monster).

For different p:

p = 100% --> factor = 50%
p = 75% --> factor = 42.6%.
p = 50% --> factor = 33.3%
p = 25% --> factor = 20%

So with a probability 50% of hitting a monster, an at-will power with half-damage on a miss will reduce the number of rounds to kill the monster by 33.3%, in comparison to the same at-will power without the half-damage on a miss part.