Comparing the Infinite Monkey Theorem (IMT) to LLMs

My prompt to ChatGPT:

I feel like LLMs are a kind of prototype or mimicry of this experiment or theory. With how LLMs are trained, in their billions or trillions of parameters, how do LLMs (in how they work) compare to the “infinite monkey theorem”?

Great — let’s dive into the math behind the entropy and sequence likelihood for both the Infinite Monkey Theorem (IMT) and Large Language Models (LLMs). This will help ground the intuitive differences we discussed in formal, measurable terms.

🔢 1. The Infinite Monkey Theorem (IMT) and Sequence Likelihood

❓Setup

Let’s assume:

• A monkey types characters uniformly at random from a keyboard of $A$ distinct characters (say, 27: 26 letters + space).
• We’re interested in the probability of a specific target sequence $S$ of length $n$.

🧮 Sequence Probability

Each character has a probability $\frac{1}{A}$. So, the probability of typing a specific sequence $S$ of length $n$ is:

Example:

Let’s take “bananas” (7 characters). With 27-character alphabet:

“I chimp, therefore I am” (24 characters, ignoring punctuation):

This probability shrinks exponentially with length, because it’s pure uniform randomness.

🧠 2. Large Language Models (LLMs) and Sequence Likelihood

LLMs estimate probabilities $P(w_i \mid w_1, \dots, w_{i-1})$ — i.e., next-token prediction.

So for a sequence $S = w_1 w_2 \dots w_n$, the joint probability is:

These conditional probabilities are non-uniform, and reflect learned structure: common sequences have much higher probability than rare or ungrammatical ones.

🔢 Relative Example:

Let’s assume that the LLM assigns these rough log probabilities (not actual values but illustrative):

Then:

Compare this to IMT:

• $P_{\text{IMT}}(S) \approx 10^{-35}$
• $P_{\text{LLM}}(S) \approx 10^{-5}$

👉 That’s a 30 orders of magnitude difference, due to learned structure.

📉 3. Entropy Perspective

❗Entropy (per symbol):

• Entropy $H$ measures the average surprise (in bits) of symbols in a distribution.

For IMT:

• Uniform over 27 characters:

• So a 100-char string has ~475 bits of entropy — all randomness.

For LLM:

• Because of prediction and structure, entropy is much lower. OpenAI has reported LLMs achieving ~1.5 to 2.5 bits/character on natural language.

So:

• LLMs: 100-char sentence ≈ 150—250 bits.
• IMT: 100-char sentence = 475 bits.

Interpretation:

• Lower entropy means less surprise: LLMs compress and reuse structure.
• IMT entropy is maximal: every character is a coin flip, no structure.

📚 4. Compression & Kolmogorov Complexity

There’s a related idea: the Kolmogorov complexity of a string — the length of the shortest program that can generate it.

• Shakespeare’s Hamlet has high surface complexity, but low algorithmic complexity — it’s full of patterns.
• LLMs effectively learn a compressed representation of those patterns.
• IMT doesn’t compress at all — it “searches” exhaustively.

So, in terms of compressibility:

🧮 Final Takeaway Table

🧠 So What Does This Mean?

LLMs are not monkeys with fast fingers. They’re more like statistical language compressors: they infer structure and use that to maximize likelihood of plausible sequences. IMT is brute-force search in an exponentially vast space with no shortcuts.

The monkey has no memory. The LLM is memory—refined into a predictive engine.

Great — let’s dive into the math behind the entropy and sequence likelihood for both the Infinite Monkey Theorem (IMT) and Large Language Models (LLMs). This will help ground the intuitive differences we discussed in formal, measurable terms.

🔢 1. The Infinite Monkey Theorem (IMT) and Sequence Likelihood

❓Setup

Let’s assume:

• A monkey types characters uniformly at random from a keyboard of $A$ distinct characters (say, 27: 26 letters + space).
• We’re interested in the probability of a specific target sequence $S$ of length $n$.

🧮 Sequence Probability

Each character has a probability $\frac{1}{A}$. So, the probability of typing a specific sequence $S$ of length $n$ is:

Example:

Let’s take “bananas” (7 characters). With 27-character alphabet:

“I chimp, therefore I am” (24 characters, ignoring punctuation):

This probability shrinks exponentially with length, because it’s pure uniform randomness.

🧠 2. Large Language Models (LLMs) and Sequence Likelihood

LLMs estimate probabilities $P(w_i \mid w_1, \dots, w_{i-1})$ — i.e., next-token prediction.

So for a sequence $S = w_1 w_2 \dots w_n$, the joint probability is:

These conditional probabilities are non-uniform, and reflect learned structure: common sequences have much higher probability than rare or ungrammatical ones.

🔢 Relative Example:

Let’s assume that the LLM assigns these rough log probabilities (not actual values but illustrative):

Then:

Compare this to IMT:

• $P_{\text{IMT}}(S) \approx 10^{-35}$
• $P_{\text{LLM}}(S) \approx 10^{-5}$

👉 That’s a 30 orders of magnitude difference, due to learned structure.

📉 3. Entropy Perspective

❗Entropy (per symbol):

• Entropy $H$ measures the average surprise (in bits) of symbols in a distribution.

For IMT:

• Uniform over 27 characters:

• So a 100-char string has ~475 bits of entropy — all randomness.

For LLM:

• Because of prediction and structure, entropy is much lower. OpenAI has reported LLMs achieving ~1.5 to 2.5 bits/character on natural language.

So:

• LLMs: 100-char sentence ≈ 150—250 bits.
• IMT: 100-char sentence = 475 bits.

Interpretation:

• Lower entropy means less surprise: LLMs compress and reuse structure.
• IMT entropy is maximal: every character is a coin flip, no structure.

📚 4. Compression & Kolmogorov Complexity

There’s a related idea: the Kolmogorov complexity of a string — the length of the shortest program that can generate it.

• Shakespeare’s Hamlet has high surface complexity, but low algorithmic complexity — it’s full of patterns.
• LLMs effectively learn a compressed representation of those patterns.
• IMT doesn’t compress at all — it “searches” exhaustively.

So, in terms of compressibility:

🧮 Final Takeaway Table

Conclusion

LLMs are not monkeys with fast fingers. They’re more like statistical language compressors: they infer structure and use that to maximize likelihood of plausible sequences. IMT is brute-force search in an exponentially vast space with no shortcuts.

The monkey has no memory. The LLM is memory—refined into a predictive engine.

Sources

Wikipedia