Review: "Minimum Message Length: A 1960s Theory of Learning for the 21st Century"

This is an elegant, readable, and intellectually generous essay. Its central thesis is clearly stated and genuinely interesting to a general statistical readership: complexity is a property of distinguishable explanations, not of coordinates. Sections 2 to 5 form a well-built argumentative spine. The prose is unusually clear for a paper on this topic. The redundant normal-mean example ($\mu = a + b$) is the kind of simple, memorable device that makes a perspective piece succeed.

The manuscript is, however, visibly a draft. Two citation placeholders remain in Section 6. The "modern problems" sections (6 and 7) are thinner and more assertive than the foundational sections that precede them. The paper engages too little with the existing literature that has already connected description length to neural network generalisation, flat minima, and quantisation. That literature includes work directly inspired by Wallace. The paper also leaves unaddressed the most obvious objection a statistician will raise: why state a single explanation at all, when the one-part Bayesian mixture code is shorter? Finally, the MDL section is somewhat one-sided in a way that may undermine the paper's credibility with neutral readers.

Recommendation: Major revision. The core is strong and publishable. The required revisions are mostly additive (a worked example, missing literature, a fairer MDL comparison, reconciling the flatness and singularity arguments) rather than a rethink.

The argument is coherent at its core. The logical chain runs as follows: an explanation must be stated; continuous parameters cannot be stated exactly; therefore only distinguishable hypotheses receive codewords; distinguishability is a property of distributions, not labels; therefore complexity must be invariant to labelling. Each step follows from the previous one, and the conclusion (Section 5) is earned.

Coherence weakens after Section 5. Section 6 relies on the regular MML87 approximation, which requires a non-singular Fisher information matrix. Section 7 then notes, correctly, that the models motivating the flat-minima literature are typically singular, and that MML87 "can fail badly" there. The paper does not reconcile the two. As written, the flat-minima argument is made with a tool that the next section concedes does not apply to the main case of interest. This is the most important internal tension in the manuscript (see Major Issue 4).

The current order is: 1 Intro, 2 Two-part codes, 3 Quantisation, 4 Invariance, 5 Complexity, 6 Flat minima, 7 Modern problems, 8 Paradox, 9 Finite precision, 10 MDL, 11 Foundational principle.

Several structural issues stand out:

• Section 8 is out of place. The MAP/SMML paradox follows directly from Sections 2 and 3, which already set up "two-part code with prior probabilities = MAP" and then introduce a codebook $\bar{\Theta}\subseteq\Theta$. Placing it after the modern-ML material forces the reader to backtrack. It would sit more naturally as a short subsection at the end of Section 3, or immediately after Section 4.

• Section 9 largely restates Section 3. Its key idea (finite precision is inferential, not numerical) is already stated well in Section 3 and again in the quantisation paragraph of Section 7. The one genuinely new point, that the data themselves must be stated to finite precision, is worth keeping but could be folded into Section 3 or the conclusion.

• Section 10 interrupts the closing arc. The MDL comparison is long and argumentative, and it sits between two reflective sections. Consider a single "Relation to other frameworks" section that covers MDL, AIC/BIC, PAC-Bayes, and singular learning theory together, placed before Section 11.

• Section 11 partly repeats the Introduction. This is acceptable in an essay, but some trimming is possible.

A suggested reordering: 1, 2, 3 (+ old 8 + finite-data point from old 9), 4, 5, 6, 7, 10 (expanded to "Relation to other frameworks"), 11.

Sections 1 to 5 build very well. Each section introduces exactly one new idea and uses it immediately. The "parameters are labels" section is the conceptual high point. From Section 6 onward the paper shifts from building an argument to listing consequences, and the pacing changes with it (see §4.3).

The writing is clear, precise, and largely jargon-light for the subject. Mathematical content is introduced with prose motivation before notation, which suits The American Statistician. A few stylistic habits recur:

• Overuse of a small set of connectives. "Rather," "Of course," "Importantly," "In this sense," "As such," and "This is…" appear many times, often sentence-initially. "Rather" in particular appears in almost every section and occasionally twice in a paragraph. A pass to vary or remove these would tighten the prose noticeably.

• Hedge-then-assert pattern. Many paragraphs open with a disclaimer ("The claim being made is not that…", "This is not to claim…", "Such an observation does not solve…"). Each one is reasonable on its own. Taken together, they create a defensive rhythm. Some of these could be consolidated into one framing statement in the Introduction.

• Some terms will be unfamiliar to the target audience. These include $\aleph_0$, "fibre," "pushes forward," and "luckiness function." Each is either self-explanatory in context or quickly explained, but a one-clause gloss for "fibre" and "luckiness function" would help.

The tone is warm, confident, and occasionally witty ("Two squabbling siblings"; "We never needed all those decimal places"). This suits a perspective piece. The dedication and the Monash history paragraph are handled gracefully, and the self-aware line that institutional history "would be a poor reason to take a theory seriously" is a nice touch.

The main tonal risk is in Section 10. The section title promises a light-hearted sibling rivalry, but the content is a fairly pointed critique of MDL. Phrases such as "More troubling is…" and the priority argument ("predates… by almost a full decade") read as advocacy. A neutral reader may come away feeling that the paper is partisan, which would weaken its broader claims. Either soften the section or own its argumentative purpose more explicitly.

• Sections 1–5: well paced. The Bernoulli example in Section 3 and the $\mu = a + b$ example in Section 4 arrive at the right moments.

• Section 6: good, though the transformation algebra is slightly heavy relative to its payoff; it could move to a short appendix or box.

• Section 7: rushed. Three significant topics (double descent, quantisation, singular models) each receive one or two paragraphs of largely assertive text, with no example, figure, or quantitative illustration. This is where the paper's most attention-grabbing claims live, and it is where the reader most wants evidence.

• Sections 8–9: the pace slows and the paper looks backward, which feels like a lull after Section 7.

• Section 10: long relative to its role.

• Section 11: a good, brisk close.

The paper is engaging, more so than most papers on information-theoretic inference. The title, the opening paragraph on rediscovered ideas, and the closing question are all effective hooks. The main barrier to engagement for a TAS reader is the absence of any figure or fully worked example. The paper describes the SMML codebook but never shows one.

1. The paper never shows an SMML solution. The six-coin-toss Bernoulli example in Section 3 is set up and then abandoned. For a general statistical audience, a fully worked small example would make the central idea concrete and memorable. Binomial with $n = 6$ or $n = 10$ and a uniform prior would work well; Farr and Wallace (2002) and Wallace (2005, Ch. 3) treat this case. The example should show the actual codebook, the partition of the data space into the sets $Y_{\bar\theta}$, the assertion probabilities $q(\bar\theta)$, and how the codebook coarsens or refines with n. A figure would be ideal. It would also be the natural place to illustrate $C(\bar\theta)$ from Section 5 numerically. This is the single most valuable addition the authors could make.

2. The one-part versus two-part question is not addressed. Any Bayesian reader will ask why an explanation should be stated at all. The one-part code based on the marginal $m(y)$ is always at least as short as any two-part code, and for prediction the posterior predictive is generally better than plugging in a point estimate. Wallace discusses this directly. A paper whose thesis is that "inference is the construction of a decodable explanation" needs to state why a single stated explanation is the right target, and what is lost and gained relative to the full posterior. Without this, a large part of the readership will regard MML as a lossy approximation to Bayesian inference rather than a foundation.

3. Missing literature that has already made several of the paper's modern arguments. The paper presents MML as anticipating ideas in modern ML. That claim is reasonable, but it currently omits the work that made the connection explicit, some of which was directly inspired by Wallace. Examples (to verify):

• Hochreiter & Schmidhuber (1997), "Flat Minima." This paper motivated flatness with an MDL argument, which makes its omission from Section 6 conspicuous.

• Hinton & van Camp (1993), on minimising the description length of weights. Also the bits-back coding literature, which descends from Wallace's work.

• Dinh, Pascanu, Bengio & Bengio (2017), "Sharp minima can generalize for deep nets." This is the obvious reference for the second placeholder in Section 6.

• Keskar et al. (2017) on sharp minima and large batches, for the first placeholder.

• Blier & Ollivier (2018), "The description length of deep learning models."

• PAC-Bayes and compression bounds: Dziugaite & Roy (2017); Zhou et al. (2019); Lotfi et al. (2022). These already tie generalisation to the description length of a quantised network.

• Louizos, Ullrich & Welling (2017), "Bayesian compression for deep learning." This is relevant to the quantisation paragraph.

Engaging with this work would strengthen the paper. It lets the authors say precisely what MML adds, for example invariance via the prior-weighted Fisher term rather than raw Hessian flatness, and the expected-length rather than worst-case objective. It also guards against reviewers concluding that the paper overclaims priority.

4. The flat-minima argument and the singular-models argument are in tension. Section 6 grounds MML's notion of flatness in the regular MML87 approximation, which needs a non-singular $J(\theta)$. Section 7 notes that MML87 fails at singularities, and the deep networks that motivate the flat-minima literature are highly singular, with exactly or nearly flat directions. The paper should either (a) restrict the Section 6 claim explicitly to regular, locally identifiable settings and say what SMML implies for singular flat regions, or (b) sketch how the SMML view handles a flat but singular basin, for instance how the codebook treats an entire fibre as one hypothesis. Option (b) would turn the tension into one of the paper's strongest points. Relatedly, the flat-minima literature measures sharpness with the Hessian of the training loss, not the expected Fisher information. The paper should state the link explicitly (observed information at the MLE) and note when the two diverge.

5. The invariance argument depends on the prior, and this cuts against the overparameterisation claim. The redundant-coordinates argument holds only if the prior on (a, b) pushes forward to the original prior on µ. In real overparameterised models the prior is specified on the weights, typically as an isotropic Gaussian or via initialisation and regularisation. The induced prior over distributions is therefore determined by the architecture. Redundancy does change the Bayesian experiment, and with it the SMML codebook and complexity. The honest conclusion is that complexity is not parameter count, but it is also not independent of how the model is parameterised, because parameterisation is usually how the prior is specified. This should be stated plainly. Without it, the double-descent paragraph implies more than the argument supports.

Similarly, the permutation symmetries of neural networks are finite. They reduce codelength by about $\log(k!)$ nats per layer of width k, a modest discount relative to the parameter count. The continuous rescaling symmetries (e.g., ReLU) are the more substantive source of redundancy. It would help to distinguish the two and give at least an order-of-magnitude sense of how much "free" redundancy each provides.

6. Section 5's explanation complexity $C(\bar\theta)$ is introduced but not developed. The definition is natural, and decomposition (9) is a nice observation. But the paper does not say whether $C(\bar\theta)$ is new, and it establishes none of its properties. Questions a reader will ask: Is it invariant to reparameterisation (it should be, and saying so reinforces the thesis)? How does it scale with n, and does it recover something like $\tfrac{k}{2}\log n$ in regular models? How does it relate to MDL parametric complexity, or to the log of Watanabe's learning coefficient terms in singular models? Even a few sentences, or the numerical illustration suggested in Major Issue 1, would turn this from a definition into a contribution.

7. The MDL comparison is incomplete and somewhat one-sided. The paper characterises refined MDL mainly as minimax regret and treats luckiness functions and conditioning as ad hoc repairs. Three points need acknowledging:

• Contemporary MDL (e.g., Grünwald 2007; Grünwald & Roos 2019, "Minimum description length revisited") explicitly admits Bayesian universal codes and treats luckiness NML as close to Bayes with particular priors. The gap is therefore narrower than presented.

• MML has a symmetric requirement. SMML needs a proper prior, and improper priors cause difficulties similar to infinite parametric complexity. The paper's own remark that subjectivity "has merely been displaced" applies in both directions.

• The paper's argument that a prior is more transparent than a luckiness function is defensible, but it is an argument, and it should be presented as one rather than as a resolution.

A fairer treatment would make the genuine differences (expected versus worst-case objective; point explanation versus coding distribution) more persuasive, not less.

8. Historical framing omits Solomonoff and Kolmogorov. The Introduction positions MML as predating MDL, AIC, VC theory, PAC learning, and SLT. It omits Solomonoff (1964) and Kolmogorov (1965), whose compression-based approaches to induction predate MML. Wallace & Dowe (1999) is cited, but only in passing at the end. For a paper with a historical framing, this omission will be noticed. Separately, citing Vapnik (1995) for "modern statistical learning theory" understates its age; Vapnik & Chervonenkis (1971) is the appropriate anchor, and MML still predates it.

1. Abstract. The abstract is good. Consider adding one sentence on what readers should take away for practice, since TAS readers value this.

2. §1, "Statistics has a habit of rediscovering old ideas." The examples (kernel smoothing and attention; change of variables and normalising flows) need citations, e.g. Nadaraya (1964) and Watson (1964), Vaswani et al. (2017), Rezende & Mohamed (2015). It would also help to say briefly in what sense attention resembles kernel smoothing.

3. §2, notation. "$p_\theta$ denotes the probability density" is followed by $p(y\mid\theta)$ throughout. Pick one convention or state that they are equivalent.

4. §2. The statement that minimising (3) is MAP is correct only for discrete Θ. The paragraph does say this, but the sentence placement makes it easy to misread as general. Section 8 later qualifies this further, which reinforces the case for moving Section 8 earlier.

5. §3, Bernoulli example. Quantify the indistinguishability. With $n = 6$ the standard error near 0.5 is about 0.2, which makes the point about 0.5 versus 0.498 immediately vivid.

6. §3, SMML estimator display. The displayed estimator $\hat{\theta} = \operatorname*{arg\,min}$ is unnumbered, unlike the surrounding equations. The preceding text says "the hypothesis in Θ" where it should say Θ̄.

7. §3. "Quantised MAP" is a helpful phrase, but q is not a prior in the usual sense. It is optimised jointly with Θ̄ and depends on the marginal of the data. A short clarification would prevent confusion, especially since Section 8 depends on this distinction.

8. §4. "Fibre" deserves a one-clause gloss for a general audience. It would also help to give a concrete prior on (a, b) that achieves the required pushforward (e.g., independent $N(0, s^2 / 2)$ priors on a and b induce $N(0, s^2)$ on µ). This makes the condition tangible and sets up Major Issue 5.

9. §4, final paragraph on MAP. This is an excellent explanation of MAP's lack of invariance. Consider promoting it slightly, perhaps with a one-line example such as Bernoulli θ versus log-odds.

10. §5, "all models are one parameter models." This is a deliberately provocative line, but it risks reading as a cardinality trick. Consider framing it explicitly as a reductio against parameter counting, or cutting it.

11. §5, "This of course begs the question." This is a common misuse. Use "raises the question."

12. §5, Eq. (8). The text says "the conditional mean regret of $\hat\theta$." It should say $\bar\theta$.

13. §5, Eq. (9). Show or sketch the one-line derivation, since the conclusion that SMML minimises weighted explanation complexity rests on it.

14. §6. Define $c(k)$, or at least note that it involves lattice quantisation constants $\kappa_k$. Also note that MML87 is an approximation to SMML rather than a strict asymptotic expansion of it. The $o_p(1)$ claim should either be qualified or cited.

15. §6. The transformation algebra could move to a box or appendix to keep the section's momentum.

16. §7, double descent. The claim that MML complexity "may be substantially smaller" than raw parameter count is plausible but unsupported. Cite an empirical description-length or compression-bound study (see Major Issue 3), or soften the claim further.

17. §7, quantisation. The point that optimal precision grows with n is valuable. A one-line statement of the regular-case scaling (precision of order $\frac{1}{\sqrt n}$ per parameter, i.e. about $\tfrac{1}{2}\log n$ nats each) would make it concrete. Contrasting this with post-training quantisation practice (fixed bit-widths independent of training set size) would sharpen the argument.

18. §7, singular models. "There is likely much more to be said" is honest but vague. One or two concrete open questions would give readers something to pursue.

19. §8. The claim that as discretisation becomes finer the solution "approximates the unrestricted SMML problem, under suitable regularity conditions" needs a citation or a brief argument.

20. §8, final paragraph. This paragraph on discarding genuinely distinct discrete hypotheses (e.g., subsets of regressors) is important and somewhat buried. It is also an interesting contrast with Bayesian model averaging and deserves a sentence making that contrast explicit.

21. §9. For continuous data, the dy terms cancel when comparing hypotheses on the same data, which is why they are usually ignored. Saying so would preempt the objection that this point has no practical consequence.

22. §10, priority claim. "Predates… by almost a full decade" is slightly inaccurate: 1968 to 1978 is a full decade. More importantly, priority claims tend to read as defensive in a perspective piece. Consider reducing their prominence.

23. §10, run-on sentence. "Wallace and Freeman later developed practical approximations…, Wallace's monograph presented MML as…; as such, the Wallace programme was therefore not merely…" This is a comma splice, and "as such… therefore" is redundant.

24. §10, Gaussian infinite regret. Rissanen (2000) on MDL denoising is an indirect citation for this claim. A more standard reference (e.g., Grünwald 2007, Ch. 11) would be better.

25. §10, inductive bias paragraph. This is a strong closing idea for the section, but it is underdeveloped. One concrete example (e.g., architecture choice as a coding commitment) would land it.

26. §11. The empirical risk minimisation analogy is apt. Consider also mentioning that maximum likelihood itself is routinely applied where it is intractable (e.g., latent variable models with EM), which is closer to the statistical audience.

27. Figures. There are none. TAS readers would benefit from at least two: an SMML codebook and data-space partition for a binomial example, and the $\mu = a + b$ fibre picture showing the singular direction.

28. References. Formatting is inconsistent. Author names vary between initials and full names (e.g., "C. S. Wallace and D. M. Boulton" versus "Chris Wallace and David Boulton"). Journal abbreviations vary ("Probl. Inform. Transm." versus full titles). DOIs are present for some entries and absent for others. Some entries include the month and others do not.

Items marked † may be PDF-extraction artefacts and should be checked against the typeset PDF.

• Lead with, or include early, a fully worked binomial SMML example with a figure. TAS readers are generalists, and one concrete picture of a codebook will carry the paper further than any amount of exposition.

• Add a short "Implications for practice" paragraph, for example on choosing reporting precision, model selection with discrete hypotheses, and interpreting flatness in fitted models.

• Keep the modern-ML material, but anchor it in references the statistical audience can follow up.

• Consider a short boxed glossary covering assertion, detail, codebook, regret, and fibre.

• Check the journal's word limits for its perspective or general-interest sections. The paper is currently a comfortable length, and the suggested additions could be offset by trimming Sections 9 and 11.

The authors have written a thoughtful and enjoyable essay that makes a real case for MML as a unifying view of inference rather than one criterion among many. The strongest material, on parameters as labels and complexity as distinguishability, is persuasive and well suited to a general statistical audience. The paper's ambitions in the modern-ML sections currently outrun its evidence and its engagement with prior work. Its fairness toward MDL could also be improved. With a worked example, the missing literature, and a reconciliation of the flatness and singularity arguments, this would be a valuable and widely read contribution.