Mindfire Technology

Cox's Theorem: Is Probability Theory Universal?

Tue, 21 Oct 2025 12:00:00 -0600

Most of life involves reasoning using incomplete information. Or put another way, it's about reasoning under uncertainty. Will it rain tomorrow? Should you carry an umbrella? Classical (propositional) logic handles things that are exactly true or false — but it doesn’t tell you how to reason when you’re unsure.

In our last several posts, particularly this one, we talked about how probability theory is a generalization of boolean and propositional logic. That is to say, probability theory extends regular deductive logic and includes it; at least for boolean and propositional logic. (There is still an open question on if this is also true for First Order logic.) So probability theory gives us a way to reason with incomplete or uncertain information.

But is probability theory just one possible way to reason under uncertainty? Or is it the only way to do so?

Cox’s theorem (R. T. Cox, 1946) attempts to answer that question. Put simply: if you want a system for plausible reasoning that is sensible and consistent, you are inevitably led to probability theory. Edwin T. Jaynes championed and popularized this view; see his book (Probability Theory: The Logic of Science).

If you want to dive into the originals, Cox’s paper and notes are available online:

What we Demand of a Reasoner (According to Jayne)

Jaynes summarized Cox’s intuition in three crisp requirements. Jayne calls these 'desiderata' because they aren't axioms assumed to be true, but rather desires or goals we wish to constrain ourselves to. Despite the simplicity and even apparent 'obviousness' of these 'desiderata', these are the only things we ask of a system that assigns degrees of plausibility to statements:

(I) Representation — degrees of plausibility are real numbers.
You should be able to say, “this is more plausible than that” and encode that judgment with a single number:

Everything gets a number. Any statement A has a plausibility pl(A) (think: a slider).
Negation is linked. Knowing pl(A) should tell you pl(not A) (they’re not independent).

The first requirement means you can always compare two (mutually exclusive) statements and say one is more plausible than the other, or they're equally plausible. On the surface this seems rather reasonable and almost inevitable. How else could plausibility work but via some way to ultimately decide between mutually exclusive options? And that implies there must be some way to weigh the options compared to their competitors. This isn't possible if you can't translate everything down to a single continuous value.

The second sub point seems intuitively correct because knowing how plausible a statement is, you should automatically know how plausible its negation (the opposite statement) is. Suppose you think “it will rain” is somewhat likely: pl(rain) = 0.8 (80% plausible). Then, under Cox-like rules, the plausibility of “not rain” is determined: pl(not rain) = 0.2. If you also learn “it’s cloudy” and that makes “rain” more plausible, the rules tell you precisely how to combine the old plausibilities and the new evidence — and that combination must behave like Bayesian updating (i.e., use conditional probabilities).

Despite the 'obviousness' of this desiderata, this is probably the most challenged requirement.

(II) Qualitative correspondence with common sense.
When information is crisp, the system should reduce to ordinary logic; small changes in evidence cause small changes in plausibility:

Respect logic: logically equivalent statements get the same plausibility.
Continuity: tiny changes in input → tiny changes in output.
The AND rule (decomposability): the plausibility of “A and B” should depend only on how plausible B is, and how plausible A would be if B were true.

The continuity requirement seems sensible because you wouldn't expect a tiny piece of new information to cause a massive, sudden jump in your confidence about something.

(III) Consistency.
Different, valid routes to the same conclusion must give the same answer; the rules must scale and compose:

Universality: the same rules must work in any domain.
Non-contradiction: if a conclusion can be derived more than one way, all derivations agree.
Scalability: rules that make sense for one case should still make sense for many repeated or combined cases.

Your system of plausible reasoning should be consistent like formal logic (for our purposes that means propositional logic). For example, if two statements are logically equivalent (they mean the same thing), they should always have the same plausibility. Likewise, if something is always true by definition (a "tautology"), its plausibility should be at the maximum possible value.

And the system should be universal – it shouldn't just apply to a specific type of problem or domain. You should be able to reason about anything, from unrelated events to complex scenarios, and the underlying rules should still hold. This is crucial for making the logic broadly applicable, just like the fundamental rules of mathematics or logic apply everywhere.

One Probability Logic to Rule Them All?

Cox showed — and later expositions (Jaynes, others) made rigorous — that those seemingly mild, intuitive demands uniquely determine the algebra of plausible reasoning. Up to a simple re-scaling, the plausibility numbers must obey the product rule and the sum/negation rule. After mapping to the usual 0–1 scale, those rules are exactly the familiar axioms of probability:

P(A and B | C) = P(A | B,C) × P(B | C) (product rule)
P(A | C) + P(not A | C) = 1 (negation/sum rule)

It is possible to come up with alternatives that might on the surface look a lot different. But there will always be a function that maps the results back to simple probability theory. The only alternative to this is a system that doesn't match those seemingly innocuous desiderata. (Which seems rather undesirable.)

In short: If you accept Jaynes desiderata, probability theory is the only consistent extension of Boolean logic to uncertain situations. Cox and Jayne argue that probability theory is not merely a tool for frequencies or gambling — it’s the uniquely rational way to handle degrees of belief in uncertain situations.

So Why Does This Matter?

Cox's theorem is used to argue for several important points:

Foundational justification for Bayesian reasoning. Cox gives a principled reason to treat probabilities as degrees of belief (not only long-run frequencies).
Objectivity and consistency. If you accept the desiderata, then any other scheme will give contradictory or “nonsensical” answers in some cases.
Applies to single events. You don’t need repeatable trials — probability applies to unique hypotheses (e.g., “life exists on other planets”) because it codifies consistent belief, not frequency.
Practical payoff. The rules Cox forces on us are exactly the ones used in Bayesian inference, decision theory, and much of modern machine learning.

Conclusion and A Quick Caution

There are ongoing technical discussions (and a few edge-case counterexamples) in the literature about which exact axioms are needed. There are also a few (not so well known) alternative theories of plausibility that try out different desiderata -- such as not requiring plausibilities to be put in terms of a single real value.

Still — this seems like a big breakthrough. It takes the idea we previously proved — that probability theory is an extension to propositional logic — and formalizes it and proves that under seemingly reasonable assumptions probability theory is the sole and only way to represent logical plausibilities.

It is not an accident that Bayesian probabilities have come to dominate Machine Learning in recent years.

From Certainty to Belief: How Probability Extends Logic - Part 1

Tue, 30 Sep 2025 09:00:00 -0600

In our previous post, we gave an introduction to probability theory. In an even earlier post, we went over the basics of propositional logic — starting with the logic of Aristotle, moving on to Boolean logic, and finally showing the link to propositional logic.

But is there any relationship between probability theory and deductive logic? In fact, there is! Boolean logic and propositional logic can be thought of as a special case of probability theory.

This may seem quite strange at first. When we think about reasoning, our minds often jump to logic: if certain statements are true, then other statements must be true. This is the bedrock of deductive logic — a system built on clear-cut, binary truths: something is either true or false.

What could logic possibly have to do with something as squishy and uncertain as probability theory? Think back to the examples we used in the previous post on probability theory. If you roll a die, you don't know for certain the outcome. But you do know that rolling two fair dice and scoring a total of 12 has only a 1 in 36 chance of happening. So if you needed to make a rational decision based on those odds, you could calculate the rationally correct choice — even though there’s no certainty like there is in deductive logic.

So just intuitively, there does seem to be something at least similar between deductive logic and probability theory. Both can be used to make rational decisions.

Given this intuition, could it be possible that there is some sort of formal relationship between deductive logic and probability theory? Or is this intuition misguided?

Deductive Logic: The World of Absolutes

To explore this further, let's start with traditional deductive logic (either propositional or Boolean logic works here), where statements are assigned one of two values: True or False. If "A implies B" is true, and "A is true" is true, then "B is true" must logically follow. There’s no room for "A is probably true" or "B is sometimes true." It’s a system of absolute certainties.

But our real world is rarely so black and white. We encounter situations where information is incomplete, observations are noisy, and conclusions are tentative. This is where probability theory offers a richer language that allows more flexible reasoning.

Probability Theory: Quantifying Uncertainty

Probability theory provides a framework for managing uncertainty. When you roll a fair die, you don't know what outcome you'll get — but you do have a rational explanation for why a 12 on two fair dice will only happen 1 in 36 throws. So you can assign a confidence to the outcome even though it’s uncertain.

In fact, this can also be true for a non-random event. In our previous post, we considered the “probability” of an asteroid hitting the Earth and even put a number on that event despite it being non-random. Re-imagining probability theory as a sort of plausibility calculus (more on this in future posts) allows us to use the same mathematical rigor for both uncertainty due to randomness in nature and uncertainty due to our ignorance.

Imagine an extension to propositional logic where, instead of only restricting yourself to the two values of 1 = True and 0 = False, you allow any continuous value between 0 and 1. If the value is 0.5, you have no reason to prefer assuming the event in question is true or false (say, you don’t know if a giant asteroid will hit the Earth this year). If the value is 1, you currently accept the event as true (you’ve observed an asteroid heading toward Earth and calculated it’s on a collision course). If the value is 0, you currently accept the event as false (you’ve scanned the sky and there are no asteroids on a collision course this year).

Now say you have no evidence to work with other than the fact that asteroids hit the Earth once every 500,000 years (as discussed in the previous post ). Then your best estimate is that there is a 0.0002% chance that a giant asteroid will hit the Earth this year.

Notice how allowing continuous values from 0 to 1 lets us mathematically express these intuitive ideas about the “probability” of a giant asteroid smashing into the Earth given some set of observations. This is true even though giant asteroids are not random events.

What, then, are we expressing if not the probability of a random event? There is considerable debate on this point, but the most obvious answer is that we’re expressing the plausibility of an event or statement being true given what we currently know — and what we currently don’t know. This is sometimes called, by Bayesians, degrees of belief (a term I don’t object to, even though I think Bayesians only have part of the truth about probability theory).

When viewed this way, it seems intuitive that probability theory is an extension of deductive logic. Deductive logic is just a special case of probability theory that allows only True (1) and False (0), whereas probability theory allows any value between 0 and 1.

Probability Theory as an Extension of Deductive Logic

Can we show formally that our intuition is correct — that deductive logic is a special case of probability theory, and that probability theory extends deductive logic?

In my next post, I’ll go over two excellent examples from David Barber’s Bayesian Reasoning and Machine Learning that demonstrate exactly that.

Dice Rolls, Coin Flips, and Death By Asteroid: A Probability Refresher

Fri, 26 Sep 2025 12:00:00 -0600

Introduction: Randomness, Belief, and Asteroids

Imagine flipping a coin or rolling a die—those feel truly random. Every flip and roll is fresh, unpredictable, and fair. When we say there is a 50% chance of getting heads or “1‑in‑6” odds of rolling a six, probability captures the genuine randomness of the experiment.

But what about something like Earth getting hit by a giant asteroid? That event doesn’t feel random in the same way—it’s a unique, one‑off cosmic occurrence based on deterministic laws. There is either an asteroid heading to earth that is going to hit us this year or there isn't. So where does probability come into play?

When we assign a probability to it, we’re really modeling our uncertainty—our lack of exact knowledge—not a repeatable random process like a coin flip.

For example, consider asteroids around 1 km in diameter—the kinds that could trigger widespread devastation or even global catastrophe. According to the Catalina Sky Survey, such an object strikes Earth on average once every 500,000 years (Catalina Sky Survey FAQ). That translates to an annual probability of approximately:

1 / 500,000 = 0.000002 (or 0.0002%)

In this case, probability is less about dice‑rolling-style pure randomness and more about representing our ignorance and expectations based on long‑term historical frequencies.

In the sections that follow, we’ll revisit classic probability concepts using coin flips and dice rolls—but we’ll also carry forward that perspective: probability is sometimes about real randomness, and sometimes about modeling how much we don’t know.

Variables and States

Let’s start with a simple example: flipping a fair coin. A coin can be in one of two states, either heads or tails.

We can represent this experiment using a random variable. Let’s call it c, for coin. The set of values that c can take is called its domain, and we write this as:

dom(c) = {heads, tails}

This means that the variable c can be in one of two states: heads or tails.

If the coin is fair, we assign equal probabilities to each outcome:

p(c = heads) = 0.5
p(c = tails) = 0.5

So when we say something like p(heads) = 0.5, we’re using shorthand for p(c = heads)— the probability that the coin lands on heads.

The important thing here is that we’re treating the coin as a variable (c), and the results it can take on (heads or tails) as states of that variable.

An event is an expression about a random variable such as two heads in a row or two heads out of three flips.

Summing Over Variables

Another important rule in probability is that the total probability over all possible states of a variable must add up to 1. This is called the normalisation condition. When you flip a coin you know that it must either land on heads or tails, so the total probability of both combined must add up to 1.0, meaning there is a 100% chance (certainty) it will be one of the two sides.

For our coin variable c, this means:

p(c = heads) + p(c = tails) = 1

Or, using more compact notation:

∑ p(c) = 1

This just says that when we consider all possible outcomes of the coin flip, one of them must happen. Either it lands heads, or it lands tails—there are no other options.

This idea of summing over all the states of a variable will show up again later when we start working with multiple variables.

In summary:

A variable represents an uncertain quantity—like the result of a coin flip.
The states of that variable are the specific outcomes it can take—heads or tails.
The domain is the full set of those possible states.
And the sum of the probabilities of all states of the variable must equal 1.

Interacting Variables: ANDs and ORs

Sometimes we’re interested in how two variables—or two events—relate to each other. One of the core rules in probability helps us figure out the chance of either event happening:

p(x or y) = p(x) + p(y) - p(x and y)

This accounts for the fact that if both events can happen together, we shouldn’t double-count that overlap.

Let’s look at an example with a 6-sided die.

Let Event A be: the die shows an even number → {2, 4, 6}
Let Event B be: the die shows a number greater than 3 → {4, 5, 6}

These events overlap at 4 and 6.

Now let’s calculate the probabilities:

p(A) = 3/6 = 0.5 (since 2, 4, and 6 are even)
p(B) = 3/6 = 0.5 (since 4, 5, and 6 are greater than 3)
p(A and B) = 2/6 ≈ 0.333 (since 4 and 6 satisfy both)

Using the formula:

p(A or B) = p(A) + p(B) - p(A and B)
p(A or B) = 0.5 + 0.5 - 0.333 = 0.667

So there’s about a 66.7% chance that the die roll is either even, or greater than 3—or both.

This rule becomes especially useful when we work with more complex combinations of events later on.

Marginal Probability

Now that we've introduced how to handle two variables at once, it might be helpful to see how to take multiple variables and reduce back to one by summing the other variables. This is called calculating the marginal probability.

Let’s say we flip two coins. We’ll call them:

c1: the result of the first flip
c2: the result of the second flip

Each coin has two possible outcomes: heads or tails.

The full joint distribution (probabilities of combined events) includes all possible combinations:

p(c1 = heads, c2 = heads) = 0.25
p(c1 = heads, c2 = tails) = 0.25
p(c1 = tails, c2 = heads) = 0.25
p(c1 = tails, c2 = tails) = 0.25

These probabilities all add up to 1, as expected.

Now let’s say we only care about the result of the first coin. We can compute its marginal probability by summing over all outcomes of the second coin:

p(c1 = heads) = p(c1 = heads, c2 = heads) + p(c1 = heads, c2 = tails)
p(c1 = heads) = 0.25 + 0.25 = 0.5

p(c1 = tails) = p(c1 = tails, c2 = heads) + p(c1 = tails, c2 = tails)
p(c1 = tails) = 0.25 + 0.25 = 0.5

This process is called marginalisation. We’re taking a joint distribution of multiple variables and extracting just the part we care about—by summing over the rest.

Conditional Probability

Sometimes we don’t just want to know the probability of an event—we want to know the probability given that something else has already happened.

This brings us to conditional probability.

We write this as:

p(x | y) = p(x and y) / p(y)
or more formally:
p(x | y) = p(x, y) / p(y)

This definition tells us how to update our probability of x when we know that y is true. The numerator is the joint probability that both x and y happen, and the denominator scales it relative to how likely y is overall.

Let’s look at a simple dice example:

Event A: the die shows a 6
Event B: the die shows an even number (2, 4, or 6)

We want to know:

p(A | B) — the probability that the die shows a 6 given that it shows an even number.

Using the formula:

p(A | B) = p(A and B) / p(B)
p(A | B) = p(6) / p(even)
p(A | B) = (1/6) / (3/6) = 1/3

So if we already know that the die landed on an even number, there’s a 1 in 3 chance it was a 6.

This is the essence of conditional probability: we’re updating what we believe about one event in light of new information.

Bayes’ Rule: Deriving It Step by Step

Let’s start with something simple but powerful:

🔁 Joint probabilities are symmetric

The joint probability of two events x and y is the same regardless of their order:

p(x, y) = p(y, x)

This just says: the probability of both x and y happening is the same as the probability of both y and x happening. They describe the same outcome.

📘 Now recall the definition of conditional probability

By definition:

p(x | y) = p(x, y) / p(y)
This gives us the probability of x given that y has occurred.

Likewise:

p(y | x) = p(y, x) / p(x)
This gives the probability of y given that x has occurred.

Let's calculate p(x, y) now by multiplying both sides by p(y):

p(x,y) = p(x|y) p(y)

Note also that we could have instead done this as:

p(y, x) = p(y|x) p(x)

🔁 Substituting using the symmetry of joint probabilities

We know from earlier that:

p(x, y) = p(y, x)

So we can replace p(x, y) in the first equation with p(y | x) × p(x) (from the second equation above):

p(x | y) = p(x, y) / p(y)
p(x | y) = [p(y | x) × p(x)] / p(y)

✅ Bayes’ Rule

We’ve now arrived at the full expression:

p(x | y) = (p(y | x) × p(x)) / p(y)

This is Bayes’ Rule. It allows us to reverse the direction of a conditional probability—from p(y | x) to p(x | y)— by incorporating:

p(x): our prior belief about x (i.e. the probability of x before we have any evidence)
p(y): the overall likelihood of observing y
p(y | x): how likely y is if x is true

Symptoms and Diseases: Using Bayes’ Rule to Flip the Question

Imagine you go to the doctor because you have a particular symptom—say, a persistent cough. Naturally, you want to know: What is the probability that you have a certain disease given this symptom?

This sounds straightforward, but here’s the catch: it’s often hard to directly measure how likely someone with that symptom actually has the disease. Symptoms can be caused by many things, and collecting data on all people with that symptom and their diagnoses is complicated.

However, it’s usually easier to measure two other things:

How often people who have the disease exhibit that symptom. For example, suppose that 80% of people with disease X have a cough. This is the probability of the symptom given the disease:

p(cough | disease) = 0.8

How common the disease is in the general population. For example, maybe disease X affects 1 in 1,000 people:

p(disease) = 0.001

With those two pieces of information, and using Bayes’ Rule, we can calculate what we really want:

What is the probability of having the disease given the symptom?

p(disease | cough) = [p(cough | disease) × p(disease)] / p(cough)

But what about p(cough), the overall probability of coughing? We can estimate that by considering:

The chance of coughing if you have the disease: p(cough | disease) = 0.8
The chance of coughing if you do not have the disease: say this is p(cough | no disease) = 0.1 (10% of people cough for other reasons)
The overall chance of having the disease: p(disease) = 0.001
The chance of not having the disease: p(no disease) = 1 - 0.001 = 0.999

To compute p(cough) — the overall probability that someone has a cough — we can break it down based on whether or not they have the disease. This follows the same logic we saw earlier with marginal probability, where we summed over the possible states of a variable.

In this case, we’re summing over two mutually exclusive groups: those with the disease and those without it. So we write:

p(cough) = p(cough, disease) + p(cough, no disease)

Using the definition of conditional probability

(p(x, y) = p(x | y) × p(y)),

we can rewrite this as:

p(cough) = p(cough | disease) × p(disease) + p(cough | no disease) × p(no disease)

This is also known as the law of total probability.

Substituting the values:

p(cough) = (0.8 × 0.001) + (0.1 × 0.999) = 0.0008 + 0.0999 = 0.1007

So even though the symptom is very common (10% of people get it regardless), the contribution from the rare disease (0.08%) is small in comparison.

Finally, apply Bayes’ Rule:

p(disease | cough) = (0.8 × 0.001) ÷ 0.1007 = 0.0008 ÷ 0.1007 ≈ 0.00795

This means:

Given that you have a cough, the chance that you have the disease is about 0.8% — or roughly 8 in 1,000.

Even though the disease is quite rare (0.1%), and the symptom common (coughing can be caused by many things), Bayes’ Rule helps us update the probability of disease in light of the symptom.

Conclusions

So what does this have to do with asteroids? I'm glad you asked.

Throughout this post, we've looked at how probability helps us reason under uncertainty:

We defined variables and their possible states, like coin flips and dice rolls.
We explored how probabilities add up across all possible outcomes of a variable.
We looked at joint probabilities and how multiple variables interact.
We introduced marginal probability by summing over hidden or unobserved variables.
And we learned how Bayes’ Rule lets us flip the direction of a probability—from something easier to measure (like symptoms given disease) to something we really want to know (like disease given symptoms).

But all of this isn’t just about games of chance.

An asteroid hitting Earth isn’t like flipping a coin—it’s governed by physics. But we still use probability because we don’t know everything: we might not know the exact location of all near-Earth objects, or the precision of their orbits decades from now.

In this case, probability reflects uncertainty in our knowledge, not randomness in the universe. And yet, the same mathematical tools apply:

We can define variables like “asteroid hits Earth this year” and assign probabilities to them.
We can update those probabilities as we gather more evidence—say, spotting a new object in the sky.
And we can reason about risk using exactly the same logic we used to diagnose a cough.

The rules of probability are universal—they help us model and update beliefs whether we're rolling dice, flipping coins, diagnosing disease, or watching the skies.