Mindfire Technology

The No Free Lunch Theorem: Why No Learning Algorithm Is Universally Best

Thu, 14 May 2026 00:00:00 -0600

In my previous post on inductive bias, I ended with an open question: is there an "optimal" inductive bias -- one whose search space is universal but whose search strategy is still efficient and tractable for any problem?

Aren't humans an "optimal" general learner compared to, say, existing machine learning algorithms? So intuitively it seems like the answer should be "yes, there is an optimal learner."

As it turns out, there is a mathematical proof known as the "No Free Lunch Theorem" that proves the answer is actually "no." It is one of the most important results in the theory of optimization.

In 1997, David Wolpert and William Macready published a paper called "No Free Lunch Theorems for Optimization" that proved something remarkable: averaged over all possible problems, no optimization strategy performs better than any other. Including random search. Including random guessing. Including humans in the loop. Every strategy that gains an advantage on some class of problems pays for it with equal disadvantage on another class.

And when they say "strategy," they mean it broadly. As Ho and Pepyne put it in their accessible explanation of the theorem: "Strategies include methods involving search, adaptation, learning, voting, feedback, dynamic programming, evolution, randomization, and even humans in the loop. In short, the concept of strategy covers any method for coming up with a solution to an optimization problem. Nothing can be more general or more inclusive" (Ho and Pepyne, 2001).

This sounds absurd. We know that some algorithms work better than others in practice. How can all strategies be equally good? The answer lies in those three words: "all possible problems." Let me show you why.

A Simple Pathfinding Problem

Imagine a robot that needs to find the shortest path from point A to point D through a small network:

    B
   / \
  A   D
   \ /
    C

There are two possible routes:

Path 1: A → B → D
Path 2: A → C → D

Each path has a total distance. The robot does not know the distances in advance -- it must pick a path and find out. The goal is to pick the shorter one.

The Universe of All Possible Problems

Let us say each path's total distance can be Short (1), Medium (5), or Long (10). A "problem" is a specific assignment of distances to both paths. In the formal framework, each such assignment is a function -- labeled f0, f1, and so on -- that maps each path to a distance. Since each path can independently be 1, 5, or 10, there are 3 x 3 = 9 possible functions. Here they all are:

The P-Matrix: All 9 Possible Problems

Problem:    f0   f1   f2   f3   f4   f5   f6   f7   f8
Path 1:      1    5   10    1    5   10    1    5   10
Path 2:      1    1    1    5    5    5   10   10   10

This table is what Ho and Pepyne call the P-matrix. The rows represent the available choices. The columns represent every possible problem -- every possible assignment of distances to paths. The entries are the distances.

Most of these nine problems will never occur in the real world. Some of them might correspond to a real map. Others are pure mathematical fiction -- worlds where both paths are equally short, or where the path that looks longer on a map is actually shorter. The P-matrix does not care about physical plausibility. It enumerates everything.

Now consider two strategies:

Strategy 1: Always take Path 1.

Strategy 2: Always take Path 2.

Strategy 1 gets the Path 1 distance on every problem: 1, 5, 10, 1, 5, 10, 1, 5, 10. Total: 48.

Strategy 2 gets the Path 2 distance on every problem: 1, 1, 1, 5, 5, 5, 10, 10, 10. Total: 48.

The totals are identical. Averaged over all nine possible problems, neither strategy is better than the other.

Where Each Strategy Wins

The totals are the same, but the individual problems tell a more interesting story.

Strategy 1 wins on f3, f6, and f7 -- problems where Path 1 is shorter than Path 2. Strategy 2 wins on f1, f2, and f5 -- problems where Path 2 is shorter. They tie on f0, f4, and f8.

On the problems where Strategy 1 wins, it wins by a combined total of (5-1) + (10-1) + (10-5) = 18.

On the problems where Strategy 2 wins, it wins by a combined total of (5-1) + (10-1) + (10-5) = 18.

The gains and losses cancel perfectly. This is not a coincidence.

Why the P-Matrix Makes This Inevitable

Look at the P-matrix again. The columns enumerate every possible combination of distances. In the Path 1 row, every possible distance (1, 5, 10) appears exactly three times. The same is true for the Path 2 row.

Ho and Pepyne point out that mathematically this is a counting matrix -- a matrix whose columns count through all possible value assignments. The key property of a counting matrix is that all row sums are equal. You can verify this by inspection: both rows sum to 48. No row can have a higher total than any other when the columns enumerate every possible assignment. This is a mathematical certainty. We showed this for our small 2 x 9 matrix, but the property holds for any size. As long as the columns enumerate every possible combination of values, the matrix is a counting matrix and the row sums will always be equal.

And that is the No Free Lunch theorem. No matter what strategy you use -- no matter how sophisticated, how clever, how well-informed -- if you sum its performance across all possible problems, you get the same total as any other strategy. The P-matrix is a counting matrix, and counting matrices have equal row sums. There is no way around this.

But Real Algorithms Do Work Better

If all strategies are truly equal, why do real algorithms outperform random guessing in practice?

Because real problems are not drawn uniformly from all possible problems. The real world has structure.

Consider the A-star algorithm, one of the most effective pathfinding algorithms ever developed. A-star uses a heuristic to decide which paths to explore first. In our network, if B is geographically close to D, A-star's heuristic estimates that the path through B is likely shorter. It explores that path first.

This heuristic relies on a specific assumption about the world: that geographic proximity correlates with travel distance. In Euclidean space, this is guaranteed. If B is close to D as the crow flies, then the travel distance from B to D cannot be wildly longer than the straight-line distance. This property -- essentially the triangle inequality -- is what makes A-star's heuristic admissible, meaning it never overestimates the true remaining distance.

In the real world, this assumption holds. Roads may be winding, but a point that is one mile away as the crow flies is never a thousand miles away by road. A-star exploits this structure to prune bad paths without exploring them, which is what makes it fast and effective.

But now imagine a world with a trans-dimensional hopper -- a device that warps space so that two points that are far apart as the crow flies can have a travel distance of nearly zero--or even a physically impossible negative distance. In this world, node C might be geographically far from D, but the hopper road from C to D is absurdly short. A-star's heuristic looks at C, estimates a long remaining distance based on the straight-line measurement, and concludes "that direction is not worth exploring." It prunes the path through C -- the path that, thanks to the hopper, is actually the shortest.

A-star does not just miss the optimal path. It confidently excludes it. Its heuristic, which is so reliable in the real world, becomes actively misleading in a world where the relationship between straight-line distance and travel distance is broken. But a random search strategy -- which assigns no meaning to geographic proximity and just tries paths arbitrarily -- would have an equal chance of stumbling onto the hopper path.

This is the No Free Lunch theorem made concrete. A-star's inductive bias is the assumption that geometry is well-behaved. That assumption makes it brilliant in our world and blind in the hopper world. The P-matrix contains both kinds of worlds, and the gains and losses cancel.

The Connection to Neural Networks

The same logic applies to every learning algorithm we have discussed in this series.

Recall from the inductive bias post that Mitchell characterized backpropagation's inductive bias as "smooth interpolation between data points." A neural network trained with backpropagation assumes that the underlying function is smooth -- that nearby inputs produce nearby outputs. This is what allows it to generalize from training data to new examples.

But the NFL theorem tells us: for every smooth function where this assumption helps, there exists an anti-smooth function where it hurts by exactly the same amount. A function where nearby inputs map to wildly different outputs will fool the neural network into confidently predicting smooth transitions that do not exist. On that function, random guessing would do just as well.

The neural network's situation is identical to A-star's. Its inductive bias -- smoothness -- makes it powerful on the kinds of problems the real world actually presents. But that power comes at a cost: poor performance on problems that violate the assumption. The P-matrix contains both kinds, and the row sums are equal.

The Connection to Popper

This is Mitchell's "futility of bias-free learning" -- from our first post in this series -- generalized to all of optimization.

Mitchell showed that a single learning algorithm with no inductive bias cannot generalize at all. The NFL theorem shows something broader: not only do you need an inductive bias to generalize, but no single inductive bias is universally best. Every bias helps on some problems and hurts on others. There is no free lunch.

And this is Karl Popper's point yet again. There is no universal method of discovery. There is no algorithm that works for everything. Every act of learning, every act of optimization, every act of scientific discovery requires prior assumptions about the structure of the problem. Those assumptions are what make progress possible -- but they are also what make us fallible. For some class of problems, they necessarily fail.

The question is never whether your algorithm has assumptions. It always does. The question is whether those assumptions match the world you are actually in.

The No Free Lunch Theorem was first proved by Wolpert and Macready (1997). The P-matrix framework and counting matrix explanation is from Ho and Pepyne (2001). All references to Mitchell are from Machine Learning (McGraw-Hill, 1997).

If you need help with your Artificial Intelligence solutions, we're here to help.

What Exactly Is an Inductive Bias?

Tue, 28 Apr 2026 00:00:00 -0600

In a previous post, I walked through Tom Mitchell's proof that a learner with no prior assumptions cannot generalize at all. The conjunctive restriction on our hypothesis space was doing all the real work -- without it, the algorithm was paralyzed. I promised a follow-up that would define that concept more precisely.

Mitchell calls it the learner's inductive bias. He defines this as:

"...the inductive bias of a learner as the set of additional assumptions B sufficient to justify its inductive inferences as deductive inferences." (Mitchell, p. 44)

Here is his formal definition:

The inductive bias of a learner L is any minimal set of assertions B such that for any target concept c and corresponding training examples D, (B + D + x) deductively entails L(x, D) for all new instances x. (Mitchell, p. 44)

Compare this to Wikipedia's definition of an 'inductive bias':

The inductive bias (also known as learning bias) of a learning algorithm is the set of assumptions that the learner uses to predict outputs of given inputs that it has not encountered (link)

In plain language: the inductive bias is whatever you'd have to add to the training data so that the learner's predictions follow by pure deduction. It is the gap between what the data says and what the learner concludes. It may be explicit, or implicit.

In the previous post, we saw that the Candidate-Elimination algorithm's inductive bias was the assertion "the target concept can be expressed as a conjunction of the attributes Genre, Mood, and Pacing." Feed that assertion to a deductive theorem prover along with the training data, and you get the same output as the so-called "inductive" learning algorithm. The "induction" was deduction plus an unstated assumption. This is why Popper was actually correct that there is no 'induction' per se. The so-called 'inductive' algorithms work just fine in real life -- but what is actually happening under the hood is always equivalent to deduction once the background knowledge is taken into consideration.

This definition is not limited to one algorithm. Every learner has an inductive bias, and identifying it tells you exactly what assumptions the learner is smuggling in.

Comparing Learners by Their Bias

As Mitchell puts it:

One advantage of viewing inductive inference systems in terms of their inductive bias is that it provides a nonprocedural means of characterizing their policy for generalizing beyond the observed data (Mitchell, p. 44)

To understand what he means, consider a "Rote-Learner" algorithm. All it does is store every training example in memory, and when you ask it to classify a new instance, it looks for an exact match. If it has seen that exact instance before, it returns the stored classification. Otherwise, it refuses to classify it at all. This algorithm has no inductive bias -- and it is misleading to even call it a "learner" because it never actually learns anything. It just stores data points. It is the "unbiased learner" from the previous post, and as we saw there, it is completely useless for generalization. (Mitchell, p. 45)

Now compare this to the Candidate-Elimination algorithm from the previous post. It classifies a new instance only when every hypothesis in the version space agrees on the answer. Its inductive bias is a single assumption: the target concept is contained in the hypothesis space H, which is the space of conjunctions between attributes. (i.e. conjunction means the attributes can be "AND"ed but not "OR"ed together.) Because it has a stronger bias than the Rote-Learner, it can classify instances the Rote-Learner cannot. But the correctness of those classifications depends entirely on whether the bias is true -- whether the target concept really is in H. If it is not, the version space may collapse entirely -- or worse, the algorithm may converge on the wrong answer. (Mitchell, p. 45).

Mitchell also describes the Find-S algorithm, which only tracks the S boundary -- the most specific hypothesis consistent with the positive examples. It ignores negative examples entirely and uses that single hypothesis to classify everything: if the hypothesis covers a new instance, it predicts positive; otherwise, it predicts negative. This gives it an even stronger bias than the Candidate-Elimination algorithm. In addition to assuming the target concept is in H, it assumes that all instances are negative unless the opposite is entailed by its other knowledge (Mitchell, p. 45). Where Candidate-Elimination would say "I don't know" when the version space is split, Find-S always has an answer: negative. The advantage is that it now always gives an answer, unlike the Candidate-Elimination algorithm. But this comes at the cost of sometimes being confidently wrong. This makes it the most aggressive of the three -- and the most dependent on its assumptions being correct.

The pattern is clear: stronger bias means more generalization, but also more risk. As Mitchell notes, "more strongly biased methods make more inductive leaps, classifying a greater proportion of unseen instances" (Mitchell, p. 45). But those leaps are only as good as the assumptions that enable them.

The pattern here generalizes. All learning algorithms can be characterized in terms of their inductive bias. Some biases are categorical restrictions that completely rule out certain concepts -- like the assumption that H contains the target concept (e.g. Candidate-Elimination algorithm). Others merely express preferences, ranking some hypotheses above others -- like "prefer simpler hypotheses over complex ones." And some inductive biases are hardwired into the algorithm's design. But it is even possible to create a learner that can modify its own inductive bias. (Mitchell, p. 45).

Restriction Bias vs. Preference Bias

Not all biases work the same way. Mitchell draws an important distinction between two kinds (Mitchell, pp. 62-64).

A restriction bias limits which hypotheses the learner can consider at all. The Candidate-Elimination algorithm has a restriction bias -- it literally cannot represent hypotheses outside its hypothesis space. If the true concept is a disjunction (i.e. "OR"s) and the space only allows conjunctions ("AND"s), the algorithm will never find it. The advantage is that within its restricted space, it searches completely -- it considers every consistent hypothesis (Mitchell, p. 64).

A preference bias does not restrict the hypothesis space but instead orders it -- the learner prefers some hypotheses over others. Decision tree learners like ID3 have a preference bias. The hypothesis space of decision trees can represent any discrete-valued function -- it is univeral! But ID3 does not search that space exhaustively. It uses a greedy, top-down strategy that favors shorter trees and places the most informative attributes near the root (Mitchell, p. 62). Its inductive bias is entirely a consequence of this search strategy, not the expressiveness of its representation.

A great example of a preference bias is genetic programming. Its hypothesis space is the set of all programs that can be composed from a given set of primitives. Since nature is computable (see the Church-Turing-Deutsch thesis), this means its search space is in principle universal -- it can represent anything. There is no restriction bias at all. But it still has an inductive bias. As Mitchell notes, the performance of genetic programming depends crucially on the choice of representation and on the choice of fitness function (Mitchell, p. 266). And realistically, the space of all programs is so vast that the algorithm will only ever explore a tiny fraction of it -- mostly relatively small programs. The evolutionary search strategy -- selection, crossover, mutation -- determines which programs get explored and which get discarded. The bias is entirely in how it searches, not in what it can represent.

Mitchell is direct about which is generally better: a preference bias is typically more desirable than a restriction bias, because it allows the learner to work within a complete hypothesis space that is guaranteed to contain the target function. A restriction bias always carries the risk that you have excluded the right answer from the start (Mitchell, p. 64).

Neural Networks: A Different Kind of Bias

When Mitchell turns to neural networks trained with backpropagation, the inductive bias becomes harder to pin down -- but it does not disappear (Mitchell, pp. 104-107).

The hypothesis space is now continuous rather than discrete. As Mitchell describes it, "every possible assignment of network weights represents a syntactically distinct hypothesis that in principle can be considered by the learner" -- the hypothesis space is the n-dimensional Euclidean space of all the network's weights (Mitchell, p. 106). And the search strategy is completely different from the algorithms we have been discussing. Backpropagation is essentially a hill-climbing algorithm that uses calculus to determine which direction is downhill. It follows the slope of the error surface, taking small steps in whatever direction reduces the error most. This means it can get trapped in local minima -- valleys that are not the deepest valley -- and is only guaranteed to converge toward some local minimum, not necessarily the global one (Mitchell, p. 104).

So what is the inductive bias? Mitchell is candid that "it is difficult to characterize precisely the inductive bias of BACKPROPAGATION learning, because it depends on the interplay between the gradient descent search and the way in which the weight space spans the space of representable functions." But he offers an approximate answer: "one can roughly characterize it as smooth interpolation between data points." Given two positive training examples with no negative examples between them, backpropagation will tend to label points in between as positive as well (Mitchell, pp. 106-107).

This is still an inductive bias -- an assumption that goes beyond what the data logically entails. The data does not say anything about what lies between the training examples. The network's architecture and training procedure are assuming the answer is smooth. That assumption is what makes generalization possible, and it is also what makes generalization fallible.

Why This Matters

Every time a learning algorithm says anything about an instance it has not seen, it is going beyond what the data alone can justify. It is making an assumption. The inductive bias is that assumption, named and made explicit.

This is Popper's point. You cannot get from observations to general theories without bringing something to the table. The question is never whether you have prior assumptions -- you always do. The question is whether they are good ones.

An Optimal Inductive Bias?

This does raise an interesting question. Is there an "optimal" inductive bias? One whose search space is universal -- every possible function -- but whose search strategy is still efficient and tractable for any problem? We have seen that genetic programming and decision trees are both universal in their hypothesis space -- they can in principle represent any function. But universality alone is not enough. Both still depend on their search strategy to find the right hypothesis, and that search strategy is where the bias lives. And we have seen that stronger biases enable more generalization but risk being wrong. Is there a sweet spot? That is a question for a future post.

All page references to Mitchell are from Machine Learning (McGraw-Hill, 1997).

If you need help with your Artificial Intelligence solutions, we're here to help.

Machine Learning 101: The Key Concepts Behind Every Learning Algorithm

Fri, 10 Apr 2026 10:53:35 -0600

Machine learning textbooks have their own vocabulary. But behind the jargon lies a process that would be deeply familiar to Karl Popper: conjecture and refutation. This post is a short reference guide to the key terms from Tom Mitchell's Machine Learning -- a foundational textbook that I also draw on in my post on the futility of unbiased learning. For each term, I will give Mitchell's definition and then show how it maps onto Karl Popper's logic of scientific discovery.

To make this concrete, imagine a 19th-century physician trying to figure out what causes a mysterious illness sweeping through a city. Patients come in with various combinations of age, diet, water source, neighborhood, and occupation. Some get sick, others do not. The physician is trying to discover the underlying rule from these observations.

Here is the truth the physician does not yet know: the illness strikes patients who drink from the river and live in the low-lying district near the tannery due to contamination from the tannery. Both conditions must be present -- river drinkers in the hills stay healthy, and tannery district residents who drink from wells stay healthy. Only the combination is deadly.

This is, as Mitchell would put it, a concept learning task (Mitchell, p. 22). The physician's job is to discover that two-attribute rule from a handful of patients -- without knowing in advance how many attributes matter or which ones.

Instance Space

In machine learning, the instance space (denoted X) is the set of all possible examples the learner could encounter (Mitchell, p. 22). In our medical example, it is the set of all possible patients -- every combination of age, diet, water source, neighborhood, occupation, and so on. A young well-drinking hillside baker. An old river-drinking tannery district laborer. Every combination, whether or not the physician has actually seen such a patient.

In Popper's framework, the instance space is the set of all possible observations or experiments. It defines the scope of what the theory is about. Most of these possible observations will never actually be made. But they all matter, because a good theory must make predictions about all of them -- not just the ones we happen to have seen.

Target Concept

The target concept (denoted c) is the true rule the learner is trying to discover (Mitchell, p. 22). It is a function that correctly classifies every instance. In our example, the target concept is: a patient gets sick if and only if they drink from the river AND live in the tannery district. The target concept is unknown to the physician. The whole point of learning is to figure out what it is.

In Popper's terms, the target concept is the law of nature we are searching for. We do not have direct access to it. We can only approach it indirectly through conjecture and refutation -- proposing theories and testing them against observations. We may never know for certain that we have found it, but we can know when we have not found it, because our conjecture will be refuted by the evidence.

Hypothesis

A hypothesis (denoted h) is a candidate theory -- one possible answer to the question "what is the target concept?" (Mitchell, p. 23). The physician might conjecture "patients who drink from the river get sick" or "patients who live in the tannery district get sick" or "old patients get sick." Each of these is a hypothesis. Some are too broad, some are too narrow, and one -- "river drinkers in the tannery district get sick" -- happens to be correct. But the physician does not know that yet.

A hypothesis is exactly what Popper calls a conjecture. It is a bold guess about the structure of reality. It may be right or wrong. What matters is that it is testable -- it makes predictions that can be checked against observations.

Hypothesis Space

The hypothesis space (denoted H) is the set of all hypotheses the learner is willing to consider (Mitchell, p. 23). This is not the set of all possible explanations -- it is the set of explanations the learner can express given its representation.

It is intractable to consider every possible hypothesis, as this would be an infinite set. But this is unnecessary. We have a lot of background knowledge that lets us constrain the possible hypotheses we will consider. So typically we start with an already partially constrained hypothesis space based on our background knowledge. This is part of our own human "inductive bias."

But suppose our physician only considers single-attribute hypotheses -- "it is the water source" or "it is the neighborhood" -- then the correct two-attribute answer is not even in the hypothesis space. The physician could examine every patient in the city and still never find the answer, because the target concept cannot be expressed within the hypothesis space he is considering. The choice of hypothesis space determines what the learner can and cannot discover.

In Popper's framework, the hypothesis space corresponds to the theoretical framework within which the scientist operates. No scientist considers every conceivable theory. They work within a tradition, a paradigm, or a set of background assumptions that constrain which conjectures are even formulable. As I discussed in my post on [inductive bias][2], this constraint is not a flaw -- it is a prerequisite for learning anything at all. But it must be the right constraint, or the truth will be invisible to you.

Training Examples

Training examples (denoted D) are the actual observations available to the learner (Mitchell, p. 23). Each training example is an instance paired with its correct classification. A positive example is a patient who got sick. A negative example is a patient who stayed healthy.

Suppose the physician has seen five patients so far:

Old, river, tannery district, laborer -- sick
Young, river, tannery district, baker -- sick
Old, river, hillside, farmer -- healthy
Young, well, tannery district, tanner -- healthy
Old, well, hillside, laborer -- healthy

In Popper's framework, these are the observations against which we test our conjectures. The third patient -- a river drinker who stayed healthy -- is a falsification of the hypothesis "river water causes the illness." The fourth patient -- a tannery district resident who stayed healthy -- falsifies "living in the tannery district causes the illness." And the fifth patient -- an old laborer who stayed healthy -- falsifies "being a laborer causes the illness." Only the conjunction "river AND tannery district" survives all five observations.

As Popper argued, falsifications are where the real action is. A thousand sick river-drinking tannery residents do not prove that the combination is the cause -- they are merely consistent with it. But each negative example eliminates entire families of wrong theories in one stroke.

Concept Learning

Mitchell defines concept learning as "inferring a boolean-valued function from training examples of its input and output" (Mitchell, p. 21). That sounds narrow, but it is surprisingly general. Our physician is doing concept learning: he has patients (instances), he knows which ones got sick and which did not (positive and negative examples), and he is searching for the rule that separates them. But this is also what scientists do all the time. Any time we seek a causal explanation -- what causes this disease, what makes this material brittle, why do some stars explode and others do not -- we are doing concept learning. We have observations, we have outcomes, and we are searching for the rule.

But here is the key reframing that Popper would appreciate: Mitchell treats concept learning as a search problem. The learner is searching through a hypothesis space for a hypothesis consistent with the training data (Mitchell, p. 23). Our physician is not staring at patients and waiting for a pattern to emerge. He is starting with a space of possible theories and eliminating the ones that fail. The hypothesis "it is the water" fails when he sees a healthy river drinker in the hills. The hypothesis "it is the neighborhood" fails when he sees a healthy well drinker in the tannery district. What remains is not what the data induced but what the data failed to refute. This is exactly what Popper and Donald Campbell called "evolutionary epistemology" -- knowledge grows not by accumulation but by variation and selection. You generate candidate theories, test them against reality, and the ones that survive criticism are what remain. The data does not build the theory. The data selects among the theories.

Version Space

The version space is the set of all hypotheses in H that are consistent with the training data observed so far (Mitchell, p. 31). As training examples come in, hypotheses that are contradicted by the data get eliminated and the version space shrinks.

After the physician's first two patients (both sick, both river-drinking tannery district residents), many hypotheses are still alive: maybe it is the river, maybe it is the district, maybe it is the combination, maybe it is being a laborer. The version space is large. But after the third patient -- a healthy river drinker in the hills -- every hypothesis that blames the river alone is eliminated. After the fourth -- a healthy well drinker in the tannery district -- every hypothesis that blames the district alone is eliminated. The version space is closing in on the truth.

In Popper's terms, the version space is the set of conjectures that have survived all attempts at refutation so far. Each new observation either leaves it unchanged or shrinks it. The goal is to shrink it until only one hypothesis remains.

Consistent Hypothesis

A consistent hypothesis is one that correctly classifies every training example seen so far (Mitchell, p. 23). It has not been falsified by any observation.

After all five patients, the hypothesis "river AND tannery district" is consistent. But so are other, more specific hypotheses. Consider: we have not yet seen a river-drinking tannery district farmer. So the hypothesis "river AND tannery district AND NOT farmer" is equally consistent with everything we have observed -- maybe farmers are somehow immune. Likewise, both sick patients happened to be either laborers or bakers, so "river AND tannery district AND (laborer OR baker)" also fits the data. We simply have not seen enough patients to distinguish these hypotheses from each other.

So "consistent" does not mean "correct." It means "not yet refuted." This is precisely Popper's point about corroboration. A theory that has survived testing is corroborated, not confirmed. It has proven its mettle so far, but it remains permanently open to future refutation. The physician would need to find a river-drinking tannery district farmer to tell these hypotheses apart.

The General-to-Specific Ordering

Mitchell observes that hypotheses can be naturally ordered from general to specific (Mitchell, p. 24). A more general hypothesis classifies more instances as positive. The hypothesis "anyone who drinks from the river gets sick" is more general than "river drinkers in the tannery district get sick" -- the first covers a superset of the cases covered by the second.

This ordering has a direct Popperian interpretation. Popper argued that more general theories are more falsifiable -- they make bolder claims about the world, and therefore there are more ways they could be wrong. "All river drinkers get sick" is easier to refute than "river drinkers in the tannery district get sick," because the first makes predictions about a far wider range of patients. Popper considered this a virtue. Bolder theories, when they survive testing, tell us more about the world.

The Candidate-Elimination algorithm exploits this ordering to efficiently search the hypothesis space -- tracking the most general and most specific surviving hypotheses and using each new observation to tighten the bounds. As I described in my post on the futility of unbiased learning, this is falsification implemented as a computer program.

The Takeaway

Machine learning is not induction. It is search -- a search through a space of conjectures, guided by observations, eliminating the hypotheses that fail. The vocabulary is different, but the logic is the same logic Karl Popper described: bold conjectures, tested against experience, with the wrong ones ruthlessly discarded.

The key terms from Mitchell map cleanly onto this process:

The instance space is the domain of possible observations
The target concept is the unknown law we seek
A hypothesis is a conjecture
The hypothesis space is the set of conjectures we are willing to consider
Training examples are the observations that test our conjectures
Concept learning is the search for surviving conjectures
The version space is the set of conjectures not yet refuted
A consistent hypothesis is a conjecture that has survived all tests so far
The general-to-specific ordering reflects Popper's insight that bolder theories are more falsifiable

All page references to Mitchell are from Machine Learning (McGraw-Hill, 1997).

If you need help with your Artificial Intelligence solutions, we're here to help.

Induction is a Myth: The Futility of Unbiased Learning

Thu, 12 Mar 2026 16:55:14 -0600

Karl Popper's Disproof of Induction

Karl Popper argued that it is logically impossible to derive a general theory from specific observations. You can stare at a million data points and no universal law will ever logically follow from them. The observations are always concrete and specific; the theory is always abstract and universal. You simply cannot get from one to the other by logic alone. (See Conjectures and Refutations: The Growth of Scientific Knowledge, pp. 251-253)

Thus induction is a myth. No "inductive logic" exists. And although there exists a "logical" interpretation of the probability calculus, there is no good reason to assume that this "generalized logic" (as it may be called) is a system of "inductive logic" (Unended Quest, p. 171)

Despite such a startlingly powerful disproof, most people assumed it must be wrong. They sought after a new kind of "inductive logic" that would allow you to somehow start with only observations and generalize to a universal law. The reasoning went that there must be such an inductive logic because we do—in practice—induce general laws all the time in science. Probability theory—particularly the Bayesian interpretation—was often advanced as the missing inductive logic that would bridge the gap. When it was discovered that probability theory could be thought of as "extending deductive logic" (by which they really meant extending propositional logic, not first order logic), many became convinced they had found what they were looking for. The arrival of Cox's theorem was often interpreted as cementing this idea.

And on top of all that, wasn't it just a fact that machine learning algorithms generalize from data every day? Your streaming service watches you binge a few movies and then somehow knows what to recommend next. That's induction, right? Isn't the machine deriving a general rule from specific observations? Popper must be wrong!

Tom Mitchell's "Futility of Bias-Free Learning"

Tom Mitchell—one of the foundational figures in machine learning—has a devastating answer to this question. In his textbook Machine Learning (McGraw-Hill, 1997), he proves something that should be far better known: a learner that makes no prior assumptions about what it's looking for cannot generalize at all. Not even a little. It can memorize what it has seen, but the moment you show it something new, it has literally no rational basis for making a classification. Mitchell calls this "The Futility of Bias-Free Learning" (Mitchell, p. 42).

What makes this so interesting is that Mitchell arrives at essentially the same conclusion as Popper, but from the completely opposite direction. Popper was a philosopher arguing against inductivism. Mitchell is a computer scientist trying to make induction work. And yet they converge on the same point: you cannot generalize from observations alone. You always need something else—some set of prior assumptions—to bridge the gap.

But Mitchell goes one step further. He shows that when you do add those prior assumptions, the resulting "inductive" algorithm is actually equivalent to a deductive theorem prover. The so-called induction was deduction all along, just as Popper claimed. It just didn't look like it.

Let me walk through how this works.

A Movie Recommendation Problem

Imagine a movie streaming service trying to figure out whether you'll enjoy a given film. To keep things tractable, suppose the system describes movies using three attributes:

Genre: Action, Comedy, Drama, Sci-Fi
Mood: Light, Dark
Pacing: Fast, Slow

Each movie is some combination of these attributes, and your reaction is binary: you either enjoyed it or you didn't. The service's job is to figure out the general rule—the target concept—that explains your taste.

This isn't meant to be a realistic example, but suppose you subconsciously only enjoy fast-paced movies with a dark mood, regardless of genre. The service is trying to figure that out and then recommend other such movies to you.

Now, there are 4 x 2 x 2 = 16 possible movie descriptions in this space. And the streaming service has watched you react to five or six of them. From those data points, it needs to learn a rule that correctly predicts your reaction to all the other movies you haven't seen yet.

This is, at its core, what Mitchell calls a "concept learning" task: searching through a space of possible hypotheses for the one that fits the observed data (Mitchell, p. 23).

How Candidate-Elimination Works (a.k.a. Falsification)

The algorithm Mitchell uses to illustrate this is called the Candidate-Elimination algorithm. And despite the dry name, what it actually does is strikingly Popperian: it starts with every possible hypothesis about your taste and then systematically eliminates the ones that are contradicted by the data.

The algorithm restricts itself to hypotheses that take the form of conjunctions of attribute values. So a hypothesis might be something like "I enjoy Action movies that are Dark"—represented as (Action, Dark, ?), where the question mark means "I don't care about this attribute."

The algorithm also uses a null symbol to mean "no value is acceptable." So the hypothesis (null, null, null) means "no movie is enjoyable"—the most specific possible hypothesis that rejects everything.

The algorithm maintains two boundaries that define a range of surviving hypotheses:

The G boundary (most general hypothesis consistent with the data)
The S boundary (most specific hypothesis consistent with the data)

Everything between these two bounds is called the "version space"—in Popper's language, it is the set of hypotheses not yet refuted by the evidence (Mitchell, pp. 32-33).

Before we see any data, G starts as broad as possible and S starts as narrow as possible:

Now the training data starts coming in.

Step 1: You watch a dark, fast-paced action movie and enjoy it. This is a positive example, so the S boundary has to move—it generalizes from "no movie is enjoyable" to the most specific hypothesis that covers this movie. The G boundary doesn't need to change yet because nothing has been ruled out:

Step 2: You watch a light, slow comedy and don't enjoy it. This is where the real elimination begins. The current G boundary—(?, ?, ?) or "every movie is enjoyable"—is inconsistent with this negative example. It predicted you'd enjoy this movie, but you didn't. So (?, ?, ?) gets falsified and must be replaced.

The algorithm replaces it with the minimal specializations that exclude this negative example while still being more general than S. There are exactly three ways to do this—each one changes a single attribute to rule out the movie you disliked:

This is a critical moment. We now have three competing theories about your taste, and the data will eventually falsify two of them.

Step 3: You watch a dark, fast-paced sci-fi movie and enjoy it. Now things get interesting on both sides. On the S side, you've liked both Action and Sci-Fi, so genre can't be the deciding factor—S generalizes to (?, Dark, Fast). On the G side, the hypothesis (Action, ?, ?) predicted you'd only like Action movies, but you just liked a Sci-Fi movie. Falsified! It gets removed from G:

Two hypotheses remain in G: maybe it's about mood (?, Dark, ?), or maybe it's about pacing (?, ?, Fast). We need more data to tell them apart.

Step 4: You watch a dark, slow-paced action movie and don't enjoy it. This movie was Dark—just like the ones you enjoyed—but you disliked it. The hypothesis (?, Dark, ?) predicted you'd enjoy it, because it says dark mood is all that matters. But you didn't. Falsified! Only (?, ?, Fast) survives in G:

Now G says (?, ?, Fast)—pacing is all that matters—while S says (?, Dark, Fast)—both mood and pacing matter. They haven't converged yet. We need one more piece of evidence.

Step 5: You watch a light, fast-paced comedy and don't enjoy it. This is the final falsification. This movie was fast-paced, matching G's only remaining requirement, but you still disliked it. The difference from the movies you enjoyed? It was Light, not Dark. G must specialize mood from "?" to "Dark"—and now S and G are identical:

The version space has been squeezed from both sides until exactly one hypothesis remains:

Notice what happened here. The algorithm did not build up a theory from observations. It tore down the wrong theories. It started with a space of conjectures and refuted them using evidence. This is Popper's falsificationism implemented as a computer program. Mitchell himself frames concept learning as a search problem (Mitchell, p. 23), and as I've argued on my podcast, search is really just a form of variation and selection—which is exactly how Donald Campbell and Karl Popper described the growth of knowledge.

If the training data is error-free and the correct hypothesis is somewhere in the hypothesis space, the algorithm will converge on it. Every incorrect hypothesis gets falsified. The correct one survives.

The Catch: A Biased Hypothesis Space

But here's the problem. We restricted the hypothesis space to conjunctions of attributes—meaning the learned rule can only take the form "attribute 1 must be X and attribute 2 must be Y and attribute 3 must be Z" (where any of those can be relaxed to "any value is fine").

That works for a rule like (?, Dark, Fast)—"dark and fast-paced, any genre." But what if your actual taste is more complicated? What if you enjoy dark action movies and also light comedies—two completely different profiles with nothing in common? There is no single conjunction of attributes that captures both. Any conjunction broad enough to include dark action movies and light comedies would also include things you don't like, such as dark comedies or light action movies.

Our conjunctive hypothesis space—including hypotheses with ? (any value) and null (no value)—contains only 46 hypotheses, a tiny fraction of the 65,536 possible concepts that could be defined over our 16 movie descriptions. As Mitchell puts it, "a very biased hypothesis space indeed!" (Mitchell, p. 41).

If the correct hypothesis is not representable in the space, the algorithm will fail. It will eliminate every hypothesis, leaving the version space empty, and you'll know something has gone wrong.

So there is an obvious temptation: why not just expand the hypothesis space to include every possible hypothesis? Remove the bias entirely. Let the learner consider any conceivable pattern in the data.

The Power Set: An Unbiased Learner

Mitchell takes this idea seriously. He proposes expanding the hypothesis space to the power set of all possible instances—the set of all possible subsets (Mitchell, p. 40). This means the learner can now represent any target concept whatsoever. No more restrictions. No more bias. The learner can consider disjunctions, negations, arbitrary combinations—everything. If you like dark action movies and light comedies but nothing else, there's a hypothesis for that. If you like exactly seven specific movies for no discernible reason, there's a hypothesis for that too.

For our movie example with 16 possible movie descriptions, the power set contains 2^16—65,536 possible target concepts. Our biased conjunctive space could only represent a handful of those. The unbiased learner must now contend with all 65,536.

Problem solved, right?

Not even close. Mitchell shows that this "unbiased" learner is now completely unable to generalize beyond the observed examples (Mitchell, p. 41).

To see why, think about what the algorithm has to work with after seeing our five training examples. It knows you liked two specific movies and disliked three specific movies. In the biased version, the conjunctive restriction forced the algorithm's hand—there were only so many ways to draw the line, and most of them got falsified. But now? The hypothesis space contains every possible way to divide the 16 movies into "liked" and "disliked." And there are a staggering number of ways to do that which are perfectly consistent with our five data points.

Consider a new movie you haven't rated—say (Drama, Dark, Fast). Should the algorithm predict you'll enjoy it? In the biased version, the answer was clear: (?, Dark, Fast) covers it, so yes. But in the unbiased version, for every hypothesis in the version space that says you'll like this movie, there exists another hypothesis that is identical in every respect—agrees on all five training examples—except that it says you won't like this one (Mitchell, p. 41). Both hypotheses are equally consistent with everything the algorithm has seen.

This isn't a minor inconvenience. It's total paralysis. The version space splits exactly 50/50 on every unseen movie (Mitchell, p. 41). The algorithm can only say "I don't know" to every new movie it encounters. It has become a glorified lookup table—perfectly memorizing what it has seen, but completely powerless to predict anything it hasn't.

The reason is almost embarrassingly simple once you see it. In the biased version, the conjunctive restriction was doing most of the work. It told the algorithm: "The answer has a structure—it's a rule defined by attribute values." That assumption is what made it possible to look at a dark, fast-paced sci-fi movie and say "this is similar to the dark, fast-paced action movie you liked, so you'll probably like this too." Without that structural assumption, there is no basis for calling any two movies "similar." Each movie is just an isolated point, and knowing you liked one tells you nothing about any other.

To converge on a single final hypothesis, this unbiased learner would need to see every single one of the 16 possible movies as a training example (Mitchell, p. 41). At which point it hasn't learned anything—it's just stored your complete viewing history.

Mitchell's Conclusion: The Futility of Bias-Free Learning

Mitchell states his conclusion directly:

"A learner that makes no a priori assumptions regarding the identity of the target concept has no rational basis for classifying any unseen instances." (Mitchell, p. 42)

Read that again. It is a remarkable statement. It means that the only reason the original Candidate-Elimination algorithm could generalize at all was because it had a built-in assumption—the bias toward conjunctive hypotheses—that constrained the space of possibilities.

Mitchell calls this assumption the algorithm's "inductive bias." We'll discuss the formal definition of inductive bias in more detail in a future post. But a short version is that the inductive bias of a learner is the minimal set of additional assertions B such that the learner's classifications follow deductively from B combined with the training data (Mitchell, pp. 42-43).

That last part is the kicker. Mitchell is saying that what looks like induction is actually deduction in disguise—just as Popper claimed. The algorithm appears to be generalizing from data, but what it is really doing is deducing conclusions from the data plus an unstated set of assumptions.

The Deductive Theorem Prover

Mitchell makes this equivalence explicit with a striking thought experiment (Mitchell, pp. 43-44). Imagine two systems side by side.

On the left, the Candidate-Elimination algorithm—exactly the one we just walked through. You feed it your movie ratings and a new movie to classify. It searches the version space and outputs a prediction.

On the right, a deductive theorem prover. You feed it the same movie ratings, the same new movie, and one additional input: the explicit assertion "the target concept can be represented as a conjunction of the attributes Genre, Mood, and Pacing."

Mitchell proves that these two systems will produce identical outputs for every possible set of training examples and every possible new instance. They are functionally the same system. The only difference is that the inductive bias is implicit in the code of the learning algorithm, while it is explicit as an input to the theorem prover.

Think about what this means. When our algorithm concluded (?, Dark, Fast), it felt like it was inducing a general rule from specific examples. But Mitchell has shown that it was actually deducing a conclusion from the training data plus an unstated assumption about the structure of the answer. The assumption—"your taste can be expressed as a conjunction of attributes"—was baked into the algorithm's design. Make that assumption explicit, hand it to a theorem prover along with the data, and you get the same answer by pure deduction.

As Mitchell puts it, the inductive bias "exists only in the eye of us beholders. Nevertheless, it is a perfectly well-defined set of assertions" (Mitchell, p. 44).

The so-called induction was deduction all along.

What This Means

Let me be blunt about the implications.

Mitchell has shown that every "inductive" learning algorithm is, underneath, a deductive system operating on unstated assumptions. The assumptions are doing the real work. The data merely selects among the possibilities that the assumptions have already circumscribed. Without those assumptions, we saw what happens: the unbiased learner is paralyzed, unable to classify a single new instance.

This is precisely what Popper argued from the philosophy side. You cannot derive general theories from observations alone. You always need a prior theoretical framework—what Kant called imposing laws upon nature—to make sense of the data. The data doesn't speak for itself. It never has.

But here is the part that both the machine learning community and many Popperians seem to miss. Popper proved that you can't generalize from observations alone. He did not prove that you can't generalize at all. Mitchell's work shows exactly when and how generalization becomes possible: when you bring background knowledge—an inductive bias—to the table. That bias, combined with observations, lets you deduce conclusions you couldn't have reached with either one alone.

The learner doesn't start from a blank slate and induce its way to knowledge. It starts with a constrained space of possibilities and uses observations to falsify the wrong ones. That is not "induction" in the classical Baconian sense that Popper demolished. It is conjecture and refutation, running on silicon.

Bias-free learning is futile. But biased learning—learning with prior theoretical commitments—is not only possible, it is the only kind of learning there is.

This is a companion post to a podcast episode where I discuss these ideas in more depth, including how they relate to Karl Popper's epistemology and David Deutsch's interpretation of it. All page references to Mitchell are from Machine Learning (McGraw-Hill, 1997).

If you need help with your Artificial Intelligence solutions, we're here to help.