Philosophy of statistics

Statistical analysis is very important in addressing the problem of induction. Can inductive inference be formalized? What are the caveats? Can inductive inference be automated? How does machine learning work?

All knowledge is, in final analysis, history. All sciences are, in the abstract, mathematics. All judgements are, in their rationale, statistics.

– C. R. Rao1

Contents

  1. Introduction to the foundations of statistics
    1. Problem of induction
    2. Early investigators
    3. Foundations of modern statistics
    4. Pedagogy
  2. Probability and its related concepts
    1. Probability
    2. Expectation and variance
    3. Cross entropy
    4. Uncertainty
    5. Bayes’ theorem
    6. Likelihood and frequentist vs bayesian probability
    7. Curse of dimensionality
  3. Statistical models
    1. Parametric models
    2. Canonical distributions
    3. Mixture models
  4. Point estimation and confidence intervals
    1. Inverse problems
    2. Bias and variance
    3. Maximum likelihood estimation
    4. Variance of MLEs
    5. Bayesian credibility intervals
    6. Uncertainty on measuring an efficiency
    7. Examples
  5. Statistical hypothesis testing
    1. Null hypothesis significance testing
    2. Neyman-Pearson theory
    3. p-values and significance
    4. Asymptotics
    5. Student’s t-test
    6. Frequentist vs bayesian decision theory
    7. Examples
  6. Uncertainty quantification
    1. Sinervo classification of systematic uncertainties
    2. Profile likelihoods
    3. Examples of poor estimates of systematic uncertanties
  7. Statistical classification
    1. Introduction
    2. Examples
  8. Causal inference
    1. Introduction
    2. Causal models
    3. Counterfactuals
  9. Exploratory data analysis
    1. Introduction
    2. Look-elsewhere effect
    3. Archiving and data science
  10. “Statistics Wars”
    1. Introduction
    2. Likelihood principle
    3. Discussion
  11. Replication crisis
    1. Introduction
    2. p-value controversy
  12. Classical machine learning
    1. Introduction
    2. History
    3. Logistic regression
    4. Softmax regression
    5. Decision trees
    6. Clustering
  13. Deep learning
    1. Introduction
    2. Gradient descent
    3. Deep double descent
    4. Regularization
    5. Batch size vs learning rate
    6. Normalization
    7. Finetuning
    8. Computer vision
    9. Natural language processing
    10. Reinforcement learning
    11. Applications in physics
  14. Theoretical machine learning
    1. Algorithmic information theory
    2. No free lunch theorems
    3. Connectivists vs symbolicists
    4. Graphical tensor notation
    5. Universal approximation theorem
    6. Relationship to statistical mechanics
    7. Relationship to gauge theory
    8. Thermodynamics of computation
  15. Information geometry
    1. Introduction
    2. Geometric understanding of classical statistics
    3. Geometric understanding of deep learning
  16. Automation
    1. AutoML
    2. Surrogate models
    3. AutoScience
  17. Implications for the realism debate
    1. Introduction
    2. Real clusters
    3. Word meanings
  18. My thoughts
  19. Annotated bibliography
    1. Mayo, D.G. (1996). Error and the Growth of Experimental Knowledge.
    2. Cowan, G. (1998). Statistical Data Analysis.
    3. James, F. (2006). Statistical Methods in Experimental Physics.
    4. Cowan, G. et al. (2011). Asymptotic formulae for likelihood-based tests of new physics.
    5. ATLAS Collaboration. (2012). Combined search for the Standard Model Higgs boson.
    6. Cranmer, K. (2015). Practical statistics for the LHC.
    7. More articles to do
  20. Links and encyclopedia articles
    1. SEP
    2. IEP
    3. Scholarpedia
    4. Wikipedia
    5. Others
  21. References

Introduction to the foundations of statistics

Problem of induction

A key issue for the scientific method, as discussed in the previous outline, is the problem of induction. Inductive inferences are used in the scientific method to make generalizations from finite data. This introduces unique avenues of error not found in purely deductive inferences, like in logic and mathematics. Compared to deductive inferences, which are sound and necessarily follow if an argument is valid and all of its premises obtain, inductive inferences can be valid and probably (not certainly) sound, and therefore can still result in error in some cases because the support of the argument is ultimately probabilistic.

A skeptic may further probe if we are even justified in using the probabilities we use in inductive arguments. What is the probability the Sun will rise tomorrow? What kind of probabilities are reasonable?

In this outline, we sketch and explore how the mathematical theory of statistics has arisen to wrestle with the problem of induction, and how it equips us with careful ways of framing inductive arguments and notions of confidence in them.

See also:

Early investigators

The art of measuring, as precisely as possible, probabilities of things, with the goal that we would be able always to choose or follow in our judgments and actions that course, which will have been determined to be better, more satisfactory, safer or more advantageous.4

Foundations of modern statistics

Pedagogy

Probability

Probability is of epistemic interest, being in some sense a measure of inductive confidence.

TODO:

Expectation and variance

Expectation:

\[ \mathbb{E}(y) \equiv \int dx \: p(x) \: y(x) \label{eq:expectation} \]

Expectation values can be approximated with a partial sum over some data or Monte Carlo sample:

\[ \mathbb{E}(y) \approx \frac{1}{n} \sum_s^n y(x_s) \label{eq:expectation_sum} \]

The variance of a random variable, \(y\), is defined as

\[\begin{align} \mathrm{Var}(y) &\equiv \mathbb{E}((y - \mathbb{E}(y))^2) \nonumber \\ &= \mathbb{E}(y^2 - 2 \: y \: \mathbb{E}(y) + \mathbb{E}(y)^2) \nonumber \\ &= \mathbb{E}(y^2) - 2 \: \mathbb{E}(y) \: \mathbb{E}(y) + \mathbb{E}(y)^2 \nonumber \\ &= \mathbb{E}(y^2) - \mathbb{E}(y)^2 \label{eq:variance} \end{align}\]

The covariance matrix, \(\boldsymbol{V}\), of random variables \(x_i\) is

\[\begin{align} V_{ij} &= \mathrm{Cov}(x_i, x_j) \equiv \mathbb{E}[(x_i - \mathbb{E}(x_i)) \: (x_j - \mathbb{E}(x_j))] \nonumber \\ &= \mathbb{E}(x_i \: x_{j} - \mu_i \: x_j - x_i \: \mu_j + \mu_i \: \mu_j ) \nonumber \\ &= \mathbb{E}(x_i \: x_{j}) - \mu_i \: \mu_j \label{eq:covariance_matrix_indexed} \end{align}\]

\[\begin{equation} \boldsymbol{V} = \begin{pmatrix} \mathrm{Var}(x_1) & \mathrm{Cov}(x_1, x_2) & \cdots & \mathrm{Cov}(x_1, x_n) \\ \mathrm{Cov}(x_2, x_1) & \mathrm{Var}(x_2) & \cdots & \mathrm{Cov}(x_2, x_n) \\ \vdots & \vdots & \ddots & \vdots \\ \mathrm{Cov}(x_n, x_1) & \mathrm{Cov}(x_n, x_2) & \cdots & \mathrm{Var}(x_n) \end{pmatrix} \label{eq:covariance_matrix_array} \end{equation}\]

Diagonal elements of the covariance matrix are the variances of each variable.

\[ \mathrm{Cov}(x_i, x_i) = \mathrm{Var}(x_i) \]

Off-diagonal elements of a covariance matrix measure how related two variables are, linearly. Covariance can be normalized to give the correlation coefficient between variables:

\[ \mathrm{Cor}(x_i, x_j) \equiv \frac{ \mathrm{Cov}(x_i, x_j) }{ \sqrt{ \mathrm{Var}(x_i) \: \mathrm{Var}(x_j) } } \label{eq:correlation_matrix} \]

which is bounded: \(-1 \leq \mathrm{Cor}(x_i, x_j) \leq 1\).

The covariance of two random vectors is given by

\[ \boldsymbol{V} = \mathrm{Cov}(\vec{x}, \vec{y}) = \mathbb{E}(\vec{x} \: \vec{y}^{\mathsf{T}}) - \vec{\mu}_x \: \vec{\mu}_{y}^{\mathsf{T}}\label{eq:covariance_matrix_vectors} \]

Cross entropy

TODO: discuss the Shannon entropy and Kullback-Leibler (KL) divergence.33

Shannon entropy:

\[ H(p) = - \underset{x\sim{}p}{\mathbb{E}}\big[ \log p(x) \big] \label{eq:shannon_entropy} \]

Cross entropy:

\[ H(p, q) = - \underset{x\sim{}p}{\mathbb{E}}\big[ \log q(x) \big] = - \sum_{x} p(x) \: \log q(x) \label{eq:cross_entropy} \]

Kullback-Leibler (KL) divergence:

\[\begin{align} D_\mathrm{KL}(p, q) &= \underset{x\sim{}p}{\mathbb{E}}\left[ \log \left(\frac{p(x)}{q(x)}\right) \right] = \underset{x\sim{}p}{\mathbb{E}}\big[ \log p(x) - \log q(x) \big] \label{eq:kl_divergence} \\ &= - H(p) + H(p, q) \\ \end{align}\]

See also the section on logistic regression.

Uncertainty

Quantiles and standard error

TODO:

Propagation of error

Given some vector of random variables, \(\vec{x}\), with estimated means, \(\vec{\mu}\), and estimated covariance matrix, \(\boldsymbol{V}\), suppose we are concerned with estimating the variance of some variable, \(y\), that is a function of \(\vec{x}\). The variance of \(y\) is given by

\[ \sigma^2_y = \mathbb{E}(y^2) - \mathbb{E}(y)^2 \,. \]

Taylor expanding \(y(\vec{x})\) about \(x=\mu\) gives

\[ y(\vec{x}) \approx y(\vec{\mu}) + \left.\frac{\partial y}{\partial x_i}\right|_{\vec{x}=\vec{\mu}} (x_i - \mu_i) \,. \]

Therefore, to first order

\[ \mathbb{E}(y) \approx y(\vec{\mu}) \]

and

\[\begin{align} \mathbb{E}(y^2) &\approx y^2(\vec{\mu}) + 2 \, y(\vec{\mu}) \, \left.\frac{\partial y}{\partial x_i}\right|_{\vec{x}=\vec{\mu}} \mathbb{E}(x_i - \mu_i) \nonumber \\ &+ \mathbb{E}\left[ \left(\left.\frac{\partial y}{\partial x_i}\right|_{\vec{x}=\vec{\mu}}(x_i - \mu_i)\right) \left(\left.\frac{\partial y}{\partial x_j}\right|_{\vec{x}=\vec{\mu}}(x_j - \mu_j)\right) \right] \\ &= y^2(\vec{\mu}) + \, \left.\frac{\partial y}{\partial x_i}\frac{\partial y}{\partial x_j}\right|_{\vec{x}=\vec{\mu}} V_{ij} \\ \end{align}\]

TODO: clarify above, then specific examples.

See:

Bayes’ theorem

\[ P(A|B) = P(B|A) \: P(A) \: / \: P(B) \label{eq:bayes_theorem} \]

Likelihood and frequentist vs bayesian probability

\[ P(H|D) = P(D|H) \: P(H) \: / \: P(D) \label{eq:bayes_theorem_hd} \]

\[ L(\theta) = P(D|\theta) \label{eq:likelihood_def_x} \]

Fisher:

To appeal to such a result is absurd. Bayes’ theorem ought only to be used where we have in past experience, as for example in the case of probabilities and other statistical ratios, met with every admissible value with roughly equal frequency. There is no such experience in this case.38

Curse of dimensionality

Statistical models

Parametric models

Canonical distributions

Bernoulli distribution

\[ \mathrm{Ber}(k; p) = \begin{cases} p & \mathrm{if}\ k = 1 \\ 1-p & \mathrm{if}\ k = 0 \end{cases} \label{eq:bernoulli} \]

which can also be written as

\[ \mathrm{Ber}(k; p) = p^k \: (1-p)^{(1-k)} \quad \mathrm{for}\ k \in \{0, 1\} \]

or

\[ \mathrm{Ber}(k; p) = p k + (1-p)(1-k) \quad \mathrm{for}\ k \in \{0, 1\} \]

TODO: explain, another important relationship is

Figure 1: Relationships among Bernoulli, binomial, categorical, and multinomial distributions.

Normal/Gaussian distribution

\[ N(x \,|\, \mu, \sigma^2) = \frac{1}{\sqrt{2\,\pi\:\sigma^2}} \: \exp\left(\frac{-(x-\mu)^2}{2\,\sigma^2}\right) \label{eq:gaussian} \]

and in \(k\) dimensions:

\[ N(\vec{x} \,|\, \vec{\mu}, \boldsymbol{\Sigma}) = (2 \pi)^{-k/2}\:\left|\boldsymbol{\Sigma}\right|^{-1/2} \: \exp\left(\frac{-1}{2}\:(\vec{x}-\vec{\mu})^{\mathsf{T}}\:\boldsymbol{\Sigma}^{-1}\:(\vec{x}-\vec{\mu})\right) \label{eq:gaussian_k_dim} \]

where \(\boldsymbol{\Sigma}\) is the covariance matrix (defined in eq. \(\eqref{eq:covariance_matrix_indexed}\)) of the distribution.

Figure 2: Detail of a figure showing relationships among univariate distributions. See the full figure here.

Mixture models

Point estimation and confidence intervals

Inverse problems

Recall that in the context of parametric models of data, \(x_i\) the pdf of which is modeled by a function, \(f(x_i ; \theta_j)\) with parameters, \(\theta_j\). In a statistical inverse problem, the goal is to infer values of the model parameters, \(\theta_j\) given some finite set of data, \(\{x_i\}\) sampled from a probability density, \(f(x_i; \theta_j)\) that models the data reasonably well.45

Bias and variance

The bias of an estimator, \(\hat\theta\), is defined as

\[ \mathrm{Bias}(\hat{\theta}) \equiv \mathbb{E}(\hat{\theta} - \theta) = \int dx \: P(x|\theta) \: (\hat{\theta} - \theta) \label{eq:bias} \]

The mean squared error (MSE) of an estimator has a similar formula to variance (eq. \(\eqref{eq:variance}\)) except that instead of quantifying the square of the difference of the estimator and its expected value, the MSE uses the square of the difference of the estimator and the true parameter:

\[ \mathrm{MSE}(\hat{\theta}) \equiv \mathbb{E}((\hat{\theta} - \theta)^2) \label{eq:mse} \]

The MSE of an estimator can be related to its bias and its variance by the following proof:

\[\begin{align} \mathrm{MSE}(\hat{\theta}) &= \mathbb{E}(\hat{\theta}^2 - 2 \: \hat{\theta} \: \theta + \theta^2) \nonumber \\ &= \mathbb{E}(\hat{\theta}^2) - 2 \: \mathbb{E}(\hat{\theta}) \: \theta + \theta^2 \end{align}\]

noting that

\[ \mathrm{Var}(\hat{\theta}) = \mathbb{E}(\hat{\theta}^2) - \mathbb{E}(\hat{\theta})^2 \]

and

\[\begin{align} \mathrm{Bias}(\hat{\theta})^2 &= \mathbb{E}(\hat{\theta} - \theta)^2 \nonumber \\ &= \mathbb{E}(\hat{\theta})^2 - 2 \: \mathbb{E}(\hat{\theta}) \: \theta + \theta^2 \end{align}\]

we see that MSE is equivalent to

\[ \mathrm{MSE}(\hat{\theta}) = \mathrm{Var}(\hat{\theta}) + \mathrm{Bias}(\hat{\theta})^2 \label{eq:mse_variance_bias} \]

For an unbiased estimator, the MSE is the variance of the estimator.

TODO:

See also:

Maximum likelihood estimation

A maximum likelihood estimator (MLE) was first used by Fisher.47

\[\hat{\theta} \equiv \underset{\theta}{\mathrm{argmax}} \: \mathrm{log} \: L(\theta) \label{eq:mle} \]

Maximizing \(\mathrm{log} \: L(\theta)\) is equivalent to maximizing \(L(\theta)\), and the former is more convenient because for data that are independent and identically distributed (i.i.d.) the joint likelihood can be factored into a product of individual measurements:

\[ L(\theta) = \prod_i L(\theta|x_i) = \prod_i P(x_i|\theta) \]

and taking the log of the product makes it a sum:

\[ \mathrm{log} \: L(\theta) = \sum_i \mathrm{log} \: L(\theta|x_i) = \sum_i \mathrm{log} \: P(x_i|\theta) \]

Maximizing \(\mathrm{log} \: L(\theta)\) is also equivalent to minimizing \(-\mathrm{log} \: L(\theta)\), the negative log-likelihood (NLL). For distributions that are i.i.d.,

\[ \mathrm{NLL} \equiv - \log L = - \log \prod_i L_i = - \sum_i \log L_i = \sum_i \mathrm{NLL}_i \]

Invariance of likelihoods under reparametrization

See also:

Ordinary least squares

Variance of MLEs

Figure 3: Transformation of non-parabolic log-likelihood to parabolic (source: my slides, recreation of F. James (2006), p. 235).

Bayesian credibility intervals

Uncertainty on measuring an efficiency

Examples

Statistical hypothesis testing

Null hypothesis significance testing

Fisher:

[T]he null hypothesis is never proved or established, but is possibly disproved, in the course of experimentation.65

Neyman-Pearson theory

Introduction

Figure 4: TODO: ROC explainer. (Wikimedia, 2015).

See also:

Neyman-Pearson lemma

Neyman-Pearson lemma:70

For a fixed signal efficiency, \(1-\alpha\), the selection that corresponds to the lowest possible misidentification probability, \(\beta\), is given by

\[ \frac{L(H_1)}{L(H_0)} > k_{\alpha} \,, \label{eq:np-lemma} \]

where \(k_{\alpha}\) is the cut value required to achieve a type-1 error rate of \(\alpha\).

Neyman-Pearson test statistic:

\[ q_\mathrm{NP} = - 2 \ln \frac{L(H_1)}{L(H_0)} \label{eq:qnp-test-stat} \]

Profile likelihood ratio:

\[ \lambda(\mu) = \frac{ L(\mu, \hat{\theta}_\mu) }{ L(\hat{\mu}, \hat{\theta}) } \label{eq:profile-llh-ratio} \]

where \(\hat{\theta}\) is the (unconditional) maximum-likelihood estimator that maximizes \(L\), while \(\hat{\theta}_\mu\) is the conditional maximum-likelihood estimator that maximizes \(L\) for a specified signal strength, \(\mu\), and \(\theta\) as a vector includes all other parameters of interest and nuisance parameters.

Neyman construction

Cranmer: Neyman construction.

Figure 5: Neyman construction for a confidence belt for \theta (source: K. Cranmer, 2020).

TODO: fix

\[ q = - 2 \ln \frac{L(\mu\,s + b)}{L(b)} \label{eq:q0-test-stat} \]

Flip-flopping

p-values and significance

Cowan et al. define a \(p\)-value as

a probability, under assumption of \(H\), of finding data of equal or greater incompatibility with the predictions of \(H\).73

Also:

It should be emphasized that in an actual scientific context, rejecting the background-only hypothesis in a statistical sense is only part of discovering a new phenomenon. One’s degree of belief that a new process is present will depend in general on other factors as well, such as the plausibility of the new signal hypothesis and the degree to which it can describe the data. Here, however, we only consider the task of determining the \(p\)-value of the background-only hypothesis; if it is found below a specified threshold, we regard this as “discovery.”74

Uppper limits

CLs method

Asymptotics

Student’s t-test

Frequentist vs bayesian decision theory

Support for using Bayes factors:

which properly separates issues of long-run behavior from evidential strength and allows the integration of background knowledge with statistical findings.85

See also:

Examples

Uncertainty quantification

Sinervo classification of systematic uncertainties

Lyons:

In analyses involving enough data to achieve reasonable statistical accuracy, considerably more effort is devoted to assessing the systematic error than to determining the parameter of interest and its statistical error.89

Figure 6: Classification of measurement uncertainties (philosophy-in-figures.tumblr.com, 2016).

Profile likelihoods

Examples of poor estimates of systematic uncertanties

Figure 7: Demonstration of sensitivity to the jet energy scale for an alleged excess in Wjj by Tommaso Dorigo (2011) (see also: GIF).

Statistical classification

Introduction

Examples

See also:

Causal inference

Introduction

See also:

Causal models

Counterfactuals

Exploratory data analysis

Introduction

Look-elsewhere effect

Archiving and data science

“Statistics Wars”

Introduction

Cranmer:

Bayes’s theorem is a theorem, so there’s no debating it. It is not the case that Frequentists dispute whether Bayes’s theorem is true. The debate is whether the necessary probabilities exist in the first place. If one can define the joint probability \(P (A, B)\) in a frequentist way, then a Frequentist is perfectly happy using Bayes theorem. Thus, the debate starts at the very definition of probability.103

Neyman:

Without hoping to know whether each separate hypothesis is true or false, we may search for rules to govern our behaviour with regard to them, in following which we insure that, in the long run of experience, we shall not be too often wrong.104

Figure 8: From Kruschke.

Likelihood principle

O’Hagan:

The first key argument in favour of the Bayesian approach can be called the axiomatic argument. We can formulate systems of axioms of good inference, and under some persuasive axiom systems it can be proved that Bayesian inference is a consequence of adopting any of these systems… If one adopts two principles known as ancillarity and sufficiency principles, then under some statement of these principles it follows that one must adopt another known as the likelihood principle. Bayesian inference conforms to the likelihood principle whereas classical inference does not. Classical procedures regularly violate the likelihood principle or one or more of the other axioms of good inference. There are no such arguments in favour of classical inference.110

Mayo:

Likelihoods are vital to all statistical accounts, but they are often misunderstood because the data are fixed and the hypothesis varies. Likelihoods of hypotheses should not be confused with their probabilities. … [T]he same phenomenon may be perfectly predicted or explained by two rival theories; so both theories are equally likely on the data, even though they cannot both be true.116

Discussion

Lyons:

Particle Physicists tend to favor a frequentist method. This is because we really do consider that our data are representative as samples drawn according to the model we are using (decay time distributions often are exponential; the counts in repeated time intervals do follow a Poisson distribution, etc.), and hence we want to use a statistical approach that allows the data “to speak for themselves,” rather than our analysis being dominated by our assumptions and beliefs, as embodied in Bayesian priors.117

Figure 9: The major virtues and vices of Bayesian, frequentist, and likelihoodist approaches to statistical inference (gandenberger.org/research/, 2015).

Goodman:

The idea that the \(P\) value can play both of these roles is based on a fallacy: that an event can be viewed simultaneously both from a long-run and a short-run perspective. In the long-run perspective, which is error-based and deductive, we group the observed result together with other outcomes that might have occurred in hypothetical repetitions of the experiment. In the “short run” perspective, which is evidential and inductive, we try to evaluate the meaning of the observed result from a single experiment. If we could combine these perspectives, it would mean that inductive ends (drawing scientific conclusions) could be served with purely deductive methods (objective probability calculations).140

Replication crisis

Introduction

p-value controversy

[N]o isolated experiment, however significant in itself, can suffice for the experimental demonstration of any natural phenomenon; for the “one chance in a million” will undoubtedly occur, with no less and no more than its appropriate frequency, however surprised we may be that it should occur to us. In order to assert that a natural phenomenon is experimentally demonstrable we need, not an isolated record, but a reliable method of procedure. In relation to the test of significance, we may say that a phenomenon is experimentally demonstrable when we know how to conduct an experiment which will rarely fail to give us a statistically significant result.145

From “The ASA president’s task force statement on statistical significance and replicability”:

P-values are valid statistical measures that provide convenient conventions for communicating the uncertainty inherent in quantitative results. Indeed, P-values and significance tests are among the most studied and best understood statistical procedures in the statistics literature. They are important tools that have advanced science through their proper application.149

Classical machine learning

Introduction

History

See also:

Logistic regression

From a probabilistic point of view,162 logistic regression can be derived from doing maximum likelihood estimation of a vector of model parameters, \(\vec{w}\), in a dot product with the input features, \(\vec{x}\), and squashed with a logistic function that yields the probability, \(\mu\), of a Bernoulli random variable, \(y \in \{0, 1\}\).

\[ p(y | \vec{x}, \vec{w}) = \mathrm{Ber}(y | \mu(\vec{x}, \vec{w})) = \mu(\vec{x}, \vec{w})^y \: (1-\mu(\vec{x}, \vec{w}))^{(1-y)} \]

The negative log-likelihood of multiple trials is

\[\begin{align} \mathrm{NLL} &= - \sum_i \log p(y_i | \vec{x}_i, \vec{w}) \nonumber \\ &= - \sum_i \log\left( \mu(\vec{x}_i, \vec{w})^{y_i} \: (1-\mu(\vec{x}_i, \vec{w}))^{(1-y_i)} \right) \nonumber \\ &= - \sum_i \log\left( \mu_i^{y_i} \: (1-\mu_i)^{(1-y_i)} \right) \nonumber \\ &= - \sum_i \big( y_i \, \log \mu_i + (1-y_i) \log(1-\mu_i) \big) \label{eq:cross_entropy_loss0} \end{align}\]

which is the cross entropy loss. Note that the first term is non-zero only when the true target is \(y_i=1\), and similarly the second term is non-zero only when \(y_i=0\).163 Therefore, we can reparametrize the target \(y_i\) in favor of \(t_{ki}\) that is one-hot in an index \(k\) over classes.

\[ \mathrm{CEL} = \mathrm{NLL} = - \sum_i \sum_k \big( t_{ki} \, \log \mu_{ki} \big) \label{eq:cross_entropy_loss1} \]

where

\[ t_{ki} = \begin{cases} 1 & \mathrm{if}\ (k = y_i = 0)\ \mathrm{or}\ (k = y_i = 1) \\ 0 & \mathrm{otherwise} \end{cases} \]

and

\[ \mu_{ki} = \begin{cases} 1-\mu_i & \mathrm{if}\ k = 0 \\ \mu_i & \mathrm{if}\ k =1 \end{cases} \]

This readily generalizes from binary classification to classification over many classes as we will discuss more below. Note that in the sum over classes, \(k\), only one term for the true class contributes.

\[ \mathrm{CEL} = - \left. \sum_i \log \mu_{ki} \right|_{k\ \mathrm{is\ such\ that}\ y_k=1} \label{eq:cross_entropy_loss2} \]

Logistic regression uses the logit function,164 which is the logarithm of the odds—the ratio of the chance of success to failure. Let \(\mu\) be the probability of success in a Bernoulli trial, then the logit function is defined as

\[ \mathrm{logit}(\mu) \equiv \log\left(\frac{\mu}{1-\mu}\right) \label{eq:logit} \]

Logistic regression assumes that the logit function is a linear function of the explanatory variable, \(x\).

\[ \log\left(\frac{\mu}{1-\mu}\right) = \beta_0 + \beta_1 x \]

where \(\beta_0\) and \(\beta_1\) are trainable parameters. (TODO: Why would we assume this?) This can be generalized to a vector of multiple input variables, \(\vec{x}\), where the input vector has a 1 prepended to be its zeroth component in order to conveniently include the bias, \(\beta_0\), in a dot product.

\[ \vec{x} = (1, x_1, x_2, \ldots, x_n)^{\mathsf{T}}\]

\[ \vec{w} = (\beta_0, \beta_1, \beta_2, \ldots, \beta_n)^{\mathsf{T}}\]

\[ \log\left(\frac{\mu}{1-\mu}\right) = \vec{w}^{\mathsf{T}}\vec{x} \]

For the moment, let \(z \equiv \vec{w}^{\mathsf{T}}\vec{x}\). Exponentiating and solving for \(\mu\) gives

\[ \mu = \frac{ e^z }{ 1 + e^z } = \frac{ 1 }{ 1 + e^{-z} } \]

This function is called the logistic or sigmoid function.

\[ \mathrm{logistic}(z) \equiv \mathrm{sigm}(z) \equiv \frac{ 1 }{ 1 + e^{-z} } \label{eq:logistic} \]

Since we inverted the logit function by solving for \(\mu\), the inverse of the logit function is the logistic or sigmoid.

\[ \mathrm{logit}^{-1}(z) = \mathrm{logistic}(z) = \mathrm{sigm}(z) \]

And therefore,

\[ \mu = \mathrm{sigm}(z) = \mathrm{sigm}(\vec{w}^{\mathsf{T}}\vec{x}) \]

Figure 10: Logistic regression.

See also:

Softmax regression

Again, from a probabilistic point of view, we can derive the use of multi-class cross entropy loss by starting with the Bernoulli distribution, generalizing it to multiple classes (indexed by \(k\)) as

\[ p(y_k | \mu) = \mathrm{Cat}(y_k | \mu_k) = \prod_k {\mu_k}^{y_k} \label{eq:categorical_distribution} \]

which is the categorical or multinoulli distribution. The negative-log likelihood of multiple independent trials is

\[ \mathrm{NLL} = - \sum_i \log \left(\prod_k {\mu_{ki}}^{y_{ki}}\right) = - \sum_i \sum_k y_{ki} \: \log \mu_{ki} \label{eq:nll_multinomial} \]

Noting again that \(y_{ki} = 1\) only when \(k\) is the true class, and is 0 otherwise, this simplifies to eq. \(\eqref{eq:cross_entropy_loss2}\).

See also:

Decision trees

Clustering

See also:

Deep learning

Introduction

Figure 11: Raw input image is transformed into gradually higher levels of representation.

Gradient descent

\[ \hat{f} = \underset{f \in \mathcal{F}}{\mathrm{argmin}} \underset{x \sim \mathcal{X}}{\mathbb{E}} L(f, x) \]

The workhorse algorithm for optimizing (training) model parameters is gradient descent:

\[ \vec{w}[t+1] = \vec{w}[t] - \eta \frac{\partial L}{\partial \vec{w}}[t] \]

In Stochastic Gradient Descent (SGD), you chunk the training data into minibatches (AKA batches), \(\vec{x}_{bt}\), and take a gradient descent step with each minibatch:

\[ \vec{w}[t+1] = \vec{w}[t] - \frac{\eta}{m} \sum_{i=1}^m \frac{\partial L}{\partial \vec{w}}[\vec{x}_{bt}] \]

where

Deep double descent

Papers:

Blogs:

Twitter threads:

Regularization

Regularization = any change we make to the training algorithm in order to reduce the generalization error but not the training error.212

Most common regularizations:

Papers:

Batch size vs learning rate

Papers:

  1. Keskar, N.S. et al. (2016). On large-batch training for deep learning: Generalization gap and sharp minima.

[L]arge-batch methods tend to converge to sharp minimizers of the training and testing functions—and as is well known—sharp minima lead to poorer generalization. In contrast, small-batch methods consistently converge to flat minimizers, and our experiments support a commonly held view that this is due to the inherent noise in the gradient estimation.

  1. Hoffer, E. et al. (2017). Train longer, generalize better: closing the generalization gap in large batch training of neural networks.

    • \(\eta \propto \sqrt{m}\)
  2. Goyal, P. et al. (2017). Accurate large minibatch SGD: Training ImageNet in 1 hour.

    • \(\eta \propto m\)
  3. You, Y. et al. (2017). Large batch training of convolutional networks.

    • Layer-wise Adaptive Rate Scaling (LARS)
  4. You, Y. et al. (2017). ImageNet training in minutes.

    • Layer-wise Adaptive Rate Scaling (LARS)
  5. Jastrzebski, S. (2018). Three factors influencing minima in SGD.

    • \(\eta \propto m\)
  6. Smith, S.L. & Le, Q.V. (2018). A Bayesian Perspective on Generalization and Stochastic Gradient Descent.

  7. Smith, S.L. et al. (2018). Don’t decay the learning rate, increase the batch size.

    • \(m \propto \eta\)
  8. Masters, D. & Luschi, C. (2018). Revisiting small batch training for deep neural networks.

This linear scaling rule has been widely adopted, e.g., in Krizhevsky (2014), Chen et al. (2016), Bottou et al. (2016), Smith et al. (2017) and Jastrzebski et al. (2017).

On the other hand, as shown in Hoffer et al. (2017), when \(m \ll M\), the covariance matrix of the weight update \(\mathrm{Cov(\eta \Delta\theta)}\) scales linearly with the quantity \(\eta^2/m\).

This implies that, adopting the linear scaling rule, an increase in the batch size would also result in a linear increase in the covariance matrix of the weight update \(\eta \Delta\theta\). Conversely, to keep the scaling of the covariance of the weight update vector \(\eta \Delta\theta\) constant would require scaling \(\eta\) with the square root of the batch size \(m\) (Krizhevsky, 2014; Hoffer et al., 2017).

  1. Lin, T. et al. (2020). Don’t use large mini-batches, use local SGD.
    - Post-local SGD.

  2. Golmant, N. et al. (2018). On the computational inefficiency of large batch sizes for stochastic gradient descent.

Scaling the learning rate as \(\eta \propto \sqrt{m}\) attempts to keep the weight increment length statistics constant, but the distance between SGD iterates is governed more by properties of the objective function than the ratio of learning rate to batch size. This rule has also been found to be empirically sub-optimal in various problem domains. … There does not seem to be a simple training heuristic to improve large batch performance in general.

  1. McCandlish, S. et al. (2018). An empirical model of large-batch training.
    • Critical batch size
  2. Shallue, C.J. et al. (2018). Measuring the effects of data parallelism on neural network training.

In all cases, as the batch size grows, there is an initial period of perfect scaling (\(b\)-fold benefit, indicated with a dashed line on the plots) where the steps needed to achieve the error goal halves for each doubling of the batch size. However, for all problems, this is followed by a region of diminishing returns that eventually leads to a regime of maximal data parallelism where additional parallelism provides no benefit whatsoever.

  1. Jastrzebski, S. et al. (2018). Width of minima reached by stochastic gradient descent is influenced by learning rate to batch size ratio.
    • \(\eta \propto m\)

We show this experimentally in Fig. 5, where similar learning dynamics and final performance can be observed when simultaneously multiplying the learning rate and batch size by a factor up to a certain limit.

  1. You, Y. et al. (2019). Large-batch training for LSTM and beyond.
    • Warmup and use \(\eta \propto m\)

[W]e propose linear-epoch gradual-warmup approach in this paper. We call this approach Leg-Warmup (LEGW). LEGW enables a Sqrt Scaling scheme in practice and as a result we achieve much better performance than the previous Linear Scaling learning rate scheme. For the GNMT application (Seq2Seq) with LSTM, we are able to scale the batch size by a factor of 16 without losing accuracy and without tuning the hyper-parameters mentioned above.

  1. You, Y. et al. (2019). Large batch optimization for deep learning: Training BERT in 76 minutes.
    • LARS and LAMB
  2. Zhang, G. et al. (2019). Which algorithmic choices matter at which batch sizes? Insights from a Noisy Quadratic Model.

Consistent with the empirical results of Shallue et al. (2018), each optimizer shows two distinct regimes: a small-batch (stochastic) regime with perfect linear scaling, and a large-batch (deterministic) regime insensitive to batch size. We call the phase transition between these regimes the critical batch size.

  1. Li, Y., Wei, C., & Ma, T. (2019). Towards explaining the regularization effect of initial large learning rate in training neural networks.

Our analysis reveals that more SGD noise, or larger learning rate, biases the model towards learning “generalizing” kernels rather than “memorizing” kernels.

  1. Kaplan, J. et al. (2020). Scaling laws for neural language models.

  2. Jastrzebski, S. et al. (2020). The break-even point on optimization trajectories of deep neural networks.

Blogs:

Normalization

Finetuning

Computer vision

Resources:

Natural language processing

Introduction

word2vec

RNNs

Chain rule of language modeling (chain rule of probability):

\[ P(x_1, \ldots, x_T) = P(x_1, \ldots, x_{n-1}) \prod_{t=n}^{T} P(x_t | x_1 \ldots x_{t-1}) \label{eq:chain_rule_of_lm} \]

or for the whole sequence:

\[ P(x_1, \ldots, x_T) = \prod_{t=1}^{T} P(x_t | x_1 \ldots x_{t-1}) \label{eq:chain_rule_of_lm_2} \]

\[ = P(x_1) \: P(x_2 | x_1) \: P(x_3 | x_1 x_2) \: P(x_4 | x_1 x_2 x_3) \ldots \]

A language model (LM), predicts the next token given previous context. The output of the model is a vector of logits, which is given to a softmax to convert to probabilities for the next token.

\[ P(x_t | x_1 \ldots x_{t-1}) = \mathrm{softmax}\left( \mathrm{model}(x_1 \ldots x_{t-1}) \right) \]

Auto-regressive inference follows this chain rule. If done with greedy search:

\[ \hat{x}_t = \underset{x_t \in V}{\mathrm{argmax}} \: P(x_t | x_1 \ldots x_{t-1}) \label{eq:greedy_search} \]

Beam search:

Backpropagation through time (BPTT):

Neural Machine Translation (NMT):

Transformers

Figure 12: Diagram of the Transformer model (source: d2l.ai).

\[ \mathrm{attention}(Q, K, V) = \mathrm{softmax}\left(\frac{Q\, K^\intercal}{\sqrt{d_k}}\right) V \label{eq:attention} \]

Attention and Transformers
Figure 13: Diagram of the BERT model (source: peltarion.com).
Computational complexity of transformers
Efficient transformers
What comes after Transformers?

Evaluation methods

Scaling laws in NLP

Language understanding

See also:

Interpretability

Linear probes:

Reinforcement learning

Pedagogy:

Tutorials:

More:

Q-learning

AlphaZero

Regret minimization

Regret matching (RM)

Consider a game like rock-paper-scissors, where there is only one action per round. Let \(v^{t}(a)\) be the value observed when playing action \(a\) on iteration \(t\).

TODO: explain that the entire rewards vector, \(v^{t}(a)\), over \(a\) is observable after the chosen action is played.

Let a strategy, \(\sigma^t\), be a probability distribution over actions, \(a \in A\). Then the value of a strategy, \(v^{t}(\sigma^{t})\), is the expectation of its value over actions.

\[ v^{t}(\sigma^{t}) = \sum_{a \in A} \sigma^{t}(a) \: v^{t}(a) \label{eq:value_of_strategy} \]

Regret, \(R^{T}\), measures how much better some sequence of strategies, \(\sigma'\), would do compared to the chosen sequence of strategies, \(\sigma = \{\sigma^1, \sigma^2, \ldots \sigma^T\}\).

\[ R^{T} \equiv \sum_{t=1}^{T} \left( v^{t}({\sigma'}^{t}) - v^{t}(\sigma^{t}) \right) \label{eq:regret} \]

External regret, \(R^{T}(a)\), measures the regret of the chosen sequence of strategies versus a hypothetical stategy where action \(a\) is always chosen.

\[ R^{T}(a) \equiv \sum_{t=1}^{T} \left( v^{t}(a) - v^{t}(\sigma^{t}) \right) \label{eq:external_regret} \]

Regret Matching (RM) is a rule to determine the strategy for the next iteration:

\[ \sigma^{t+1}(a) \equiv \frac{ R^{t}_{+}(a) }{ \sum_{b \in A} R^{t}_{+}(b) } \label{eq:regret_matching} \]

where \(R_{+} \equiv \mathrm{max}(R, 0)\).

At the end of training, the resulting recommended strategy with convergence bounds is not the final strategy used in training, \(\sigma^{T}\), but the average strategy over all time steps:

\[ \bar{\sigma}^{T}(a) = \frac{1}{T} \sum_{t=1}^{T} \sigma^{t}(a) \]

TODO: explain the convergence of \(\bar{\sigma}^{t}\) to an \(\varepsilon\)-Nash equilibrium.

Counterfactual regret minimization (CFR)

TODO: explain extensive-form games.

A finite extensive game with imperfect information has the following components:332

The player reach, \(\pi^{\sigma}_{i}(h)\), of a history \(h\) is the product of the probabilities for all agent \(i\) actions leading to \(h\). Formally,333

\[ \pi^{\sigma}_{i}(h) \equiv \prod_{h' \cdot a' \sqsubseteq h | P(h') = i} \sigma_{i}(h', a') \label{eq:player_reach} \]

Due to perfect recall, any two histories in infoset \(I_i\) have the same player reach for player \(i\). Thus, we similarly define the player reach \(\pi^{\sigma}_{i}(I_i)\) of infoset \(I_i\) as

\[ \pi^{\sigma}_{i}(I_i) \equiv \prod_{ {I'}_{i} \cdot a' \sqsubseteq I_i | P(I_i) = i } \sigma_{i}({I'}_{i}, a') = \left.\pi^{\sigma}_{i}(h)\right|_{h \in I_i} \label{eq:player_reach_from_infoset} \]

The external reach AKA opponent reach, \(\pi^{\sigma}_{-i}(h)\), of a history \(h\) is the contribution of chance and all other players than \(i\). Formally,

\[ \pi^{\sigma}_{-i}(h) \equiv \prod_{h' \cdot a' \sqsubseteq h | P(h') \neq i} \sigma_{i}(h', a') \label{eq:external_reach} \]

We also define the external reach of an infoset as

\[ \pi^{\sigma}_{-i}(I_i) \equiv \sum_{h \in I_{i}} \pi^{\sigma}_{-i}(h) \label{eq:external_reach_from_infoset} \]

The counterfactual value of an infoset \(I\) is the expected utility to player \(i\) given that \(I\) has been reached, weighted by the external reach of \(I\) for player \(i\). Formally,334

\[ v(I) = \sum_{h \in I} \pi^{\sigma}_{-i}(h) \sum_{z \in Z} \pi^{\sigma}(h, z) \: u_{i}(z) \label{eq:counter_factual_value} \]

The counterfactual value of an action, \(a\), is

\[ v(I, a) = \sum_{h \in I} \pi^{\sigma}_{-i}(h) \sum_{z \in Z} \pi^{\sigma}(h \cdot a, z) \: u_{i}(z) \label{eq:counter_factual_value_of_a} \]

Let’s consider the case where, like in NLHE, our two private hole cards each make a single unique history \(h\), and we form infosets with a single hand, so \(I=h\). Then

\[ v(h) = \pi^{\sigma}_{-i}(h) \sum_{z \in Z} \pi^{\sigma}(h, z) \: u_{i}(z) \]

making explicit the player reach and the external reach,

\[ v(h) = \pi^{\sigma}_{-i}(h) \sum_{z \in Z} \pi_{i}^{\sigma}(h, z) \: \pi_{-i}^{\sigma}(h, z) \: u_{i}(z) \]

At a leaf node where we finally calculate the rewards,

\[ v(z) = \pi^{\sigma}_{-i}(z) \: u_{i}(z) \]

TODO: explain CFR.

The instantaneous regret is

\[ r^{t}(I, a) = v^{\sigma^t}(I, a) - v^{\sigma^t}(I) \]

The (cummulative) counterfactual regret

\[ R^{t}(I, a) = \sum_{t=1}^{T} r^{t}(I, a) \]

Similar to the single-node game discussed above, eq. \(\eqref{eq:regret_matching}\), applying regret matching during training means to update strategies according to the following rule.

\[ \sigma^{t+1}(I, a) \equiv \frac{ R^{t}_{+}(I, a) }{ \sum_{b \in A} R^{t}_{+}(I, b) } \label{eq:regret_matching_cfr} \]

The average strategy is

\[ \bar{\sigma}^{T}(I, a) = \sum_{t=1}^{T} \frac{\pi^{t}_{i}(I) \: \sigma^{t}(I, a) }{\pi^{t}_{i}(I)} \]

Monte Carlo Counterfactual Regret Minimization (MCCFR)

TODO: explain MCCFR.

External sampling MCCFR:

\[ \tilde{v}^{\sigma}_{i}(I) = \sum_{z \in Q} u_{i}(z) \: \pi^{\sigma}_{i}(z[I] \rightarrow z) \label{eq:external_sample_mccfr} \]

Best response and exploitability

Best response:

\[ \mathrm{BR}(\sigma_{-i}) = \underset{\sigma_{i}^{\prime}}{\mathrm{argmax}} \: u_{i}(\sigma_{i}^{\prime}, \sigma_{-i}) \label{eq:best_response} \]

TODO: Local Best Response (LBR).347

Exploitability:

\[ \varepsilon_{i}(\sigma) = u_{i}(\mathrm{BR}(\sigma_{-i}), \sigma_{-i}) - u_{i}(\sigma_{i}, \mathrm{BR}(\sigma_{i})) \label{eq:exploitability} \]

NashConv348 exploitability uses the convention:

\[ \varepsilon_{i}(\sigma) = u_{i}(\mathrm{BR}(\sigma_{-i}), \sigma_{-i}) - u_{i}(\sigma_{i}, \sigma_{-i}) \label{eq:nc_exploitability} \]

The average exploitability per player is

\[ \varepsilon(\sigma) = \frac{1}{n} \sum_{i}^{n} \varepsilon_{i}(\sigma) \]

Note that in zero-sum games, when summing over players, the second terms in NashConv sum to zero.349

\[ \varepsilon(\sigma) = \frac{1}{n} \sum_{i}^{n} u_{i}(\mathrm{BR}(\sigma_{-i}), \sigma_{-i}) \label{eq:average_exploitability} \]

In two-player games:

\[ \varepsilon(\sigma) = \frac{1}{2} \Big( u_{1}(\mathrm{BR}(\sigma_{2}), \sigma_{2}) + u_{2}(\sigma_{1}, \mathrm{BR}(\sigma_{1})) \Big) \label{eq:average_exploitability_two_player} \]

Solving poker

Applications in physics

See also:

Theoretical machine learning

Algorithmic information theory

No free lunch theorems

Raissi et al.:

encoding such structured information into a learning algorithm results in amplifying the information content of the data that the algorithm sees, enabling it to quickly steer itself towards the right solution and generalize well even when only a few training examples are available.395

Roberts:

From an algorithmic complexity standpoint it is somewhat miraculous that we can compress our huge look-up table of experiment/outcome into such an efficient description. In many senses, this type of compression is precisely what we mean when we say that physics enables us to understand a given phenomenon.396

Connectivists vs symbolicists

Graphical tensor notation

Universal approximation theorem

Relationship to statistical mechanics

Relationship to gauge theory

Invariant:

\[ f(g x) = f(x) \]

Equivariant:

\[ f(g x) = g' f(x) \]

Same-equivariant is the case that \(g' = g\).

Thermodynamics of computation

Information geometry

Introduction

Geometric understanding of classical statistics

Geometric understanding of deep learning

Automation

AutoML

Surrogate models

Lectures:

AutoScience

Figure 14: The inference cycle for the process of scientific inquiry. The three distinct forms of inference (abduction, deduction, and induction) facilitate an all-encompassing vision, enabling HPC and HDA to converge in a rational and structured manner. HPC: high- performance computing; HDA: high-end data analysis.

See also:

Implications for the realism debate

Introduction

See also:

Real clusters

See also:

Word meanings

Wittgenstein in PI:

The meaning of a word is its use in the language.465

and

One cannot guess how a word functions. One has to look at its use, and learn from that.466

Piantadosi:

Modern large language models integrate syntax and semantics in the underlying representations: encoding words as vectors in a high-dimensional space, without an effort to separate out e.g. part of speech categories from semantic representations, or even predict at any level of analysis other than the literal word. Part of making these models work well was in determining how to encode semantic properties into vectors, and in fact initializing word vectors via encodings of distribution semantics from e.g. Mikolov et al. 2013 (Radford et al. 2019). Thus, an assumption of the autonomy of syntax is not required to make models that predict syntactic material and may well hinder it.467

See also:

My thoughts

My docs:

My talks:

Annotated bibliography

Mayo, D.G. (1996). Error and the Growth of Experimental Knowledge.

  • Mayo (1996)

My thoughts

  • TODO

Cowan, G. (1998). Statistical Data Analysis.

  • Cowan (1998) and Cowan (2016)

My thoughts

  • TODO

James, F. (2006). Statistical Methods in Experimental Physics.

  • F. James (2006)

My thoughts

  • TODO

Cowan, G. et al. (2011). Asymptotic formulae for likelihood-based tests of new physics.

  • Cowan et al. (2011)
  • Glen Cowan, Kyle Cranmer, Eilam Gross, Ofer Vitells

My thoughts

  • TODO

ATLAS Collaboration. (2012). Combined search for the Standard Model Higgs boson.

My thoughts

  • TODO

Cranmer, K. (2015). Practical statistics for the LHC.

  • Cranmer (2015)

My thoughts

  • TODO

  • All of Statistics468
  • The Foundations of Statistics469

SEP

IEP

Scholarpedia

Wikipedia

Others

References

Agresti, A. & Coull, B. A. (1998). Approximate is better than "exact" for interval estimation of binomial proportions. The American Statistician, 52, 119–126.
Aifer, M. et al. (2023). Thermodynamic linear algebra. https://arxiv.org/abs/2308.05660
Alain, G. & Bengio, Y. (2016). Understanding intermediate layers using linear classifier probes. https://arxiv.org/abs/1610.01644
Aldrich, J. (1997). R. A. Fisher and the making of maximum likelihood 1912-1922. Statistical Science, 12, 162–176.
Amari, S. (1998). Natural gradient works efficiently in learning. Neural Computation, 10, 251–276.
———. (2016). Information Geometry and Its Applications. Springer Japan.
Anderson, C. (2008). The End of Theory: The data deluge makes the scientific method obsolete. Wired. June 23, 2008. https://www.wired.com/2008/06/pb-theory/
Andrews, M. (2023). The devil in the data: Machine learning & the theory-free ideal. https://philsci-archive.pitt.edu/22690/1/ML_Atheoreticity.pdf
Arras, K. O. (1998). An introduction to error propagation: Derivation, meaning and examples of \(C_y= F_x C_x F_{x}^{\top}\). EPFL-ASL-TR-98-01 R3. http://srl.informatik.uni-freiburg.de/papers/arrasTR98.pdf
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017). Deep Reinforcement Learning: A Brief Survey. IEEE Signal Processing Magazine, 34, 26–38.
Asch, M. et al. (2018). Big data and extreme-scale computing: Pathways to Convergence-Toward a shaping strategy for a future software and data ecosystem for scientific inquiry. The International Journal of High Performance Computing Applications, 32, 435–479.
ATLAS and CMS Collaborations. (2011). Procedure for the LHC Higgs boson search combination in Summer 2011. CMS-NOTE-2011-005, ATL-PHYS-PUB-2011-11. http://cds.cern.ch/record/1379837
ATLAS Collaboration. (2012). Combined search for the Standard Model Higgs boson in \(pp\) collisions at \(\sqrt{s}\) = 7 TeV with the ATLAS detector. Physical Review D, 86, 032003. https://arxiv.org/abs/1207.0319
ATLAS Statistics Forum. (2011). The CLs method: Information for conference speakers. http://www.pp.rhul.ac.uk/~cowan/stat/cls/CLsInfo.pdf
Aytekin, C. (2022). Neural networks are decision trees. https://arxiv.org/abs/2210.05189
Bach, F. (2022). Learning Theory from First Principles. (Draft). https://www.di.ens.fr/~fbach/ltfp_book.pdf
———. (2024). Scaling laws of optimization. https://francisbach.com/scaling-laws-of-optimization/
Bahdanau, D., Cho, K., & Bengio, Y. (2015). Neural machine translation by jointly learning to align and translate. International Conference on Learning Representations, 3rd, 2015. https://arxiv.org/abs/1409.0473
Bahri, Y. et al. (2020). Statistical mechanics of deep learning. Annual Review of Condensed Matter Physics, 11, 501–528.
Balasubramanian, V. (1996a). A geometric formulation of Occam’s razor for inference of parametric distributions. https://arxiv.org/abs/adap-org/9601001
———. (1996b). Statistical inference, Occam’s razor and statistical mechanics on the space of probability distributions. https://arxiv.org/abs/cond-mat/9601030
Balestriero, R., Pesenti, J., & LeCun, Y. (2021). Learning in high dimension always amounts to extrapolation. https://arxiv.org/abs/2110.09485
Banerjee, S., Agarwal, A., & Singla, S. (2024). LLMs will always hallucinate, and we need to live with this. https://arxiv.org/abs/2409.05746
Batson, J., Haaf, C. G., Kahn, Y., & Roberts, D. A. (2021). Topological obstructions to autoencoding. https://arxiv.org/abs/2102.08380
Battiloro, C. et al. (2024). E(n) equivariant topological neural networks. https://arxiv.org/abs/2405.15429
Baydin, A.G. et al. (2019). Etalumis: Bringing probabilistic programming to scientific simulators at scale. https://arxiv.org/abs/1907.03382
Behnke, O., Kröninger, K., Schott, G., & Schörner-Sadenius, T. (2013). Data Analysis in High Energy Physics: A Practical Guide to Statistical Methods. Wiley.
Belinkov, Y. (2022). Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48, 207–219.
Belkin, M. (2021). Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation. https://arxiv.org/abs/2105.14368
Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019). Reconciling modern machine-learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences, 116, 15849–15854. https://arxiv.org/abs/1812.11118
Bellman, R. (1952). On the theory of dynamic programming. Proceedings of the National Academy of Sciences, 38, 716–719.
Bender, E. M. & Koller, A. (2020). Climbing towards NLU: On meaning, form, and understanding in the age of data. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 58, 5185–5198. https://aclanthology.org/2020.acl-main.463.pdf
Bengio, Y. (2009). Learning deep architectures for AI. Foundations and Trends in Machine Learning, 2, 1–127. https://www.iro.umontreal.ca/~lisa/pointeurs/TR1312.pdf
Benjamin, D.J. et al. (2017). Redefine statistical significance. PsyArXiv. July 22, 2017. https://psyarxiv.com/mky9j/
Benjamini, Y. et al. (2021). The ASA president’s task force statement on statistical significance and replicability. Annals of Applied Statistics, 16, 1–2. https://magazine.amstat.org/blog/2021/08/01/task-force-statement-p-value/
Bensusan, H. (2000). Is machine learning experimental philosophy of science? In ECAI2000 Workshop notes on scientific Reasoning in Artificial Intelligence and the Philosophy of Science (pp. 9–14).
Berger, J. O. (2003). Could Fisher, Jeffreys and Neyman have agreed on testing? Statistical Science, 18, 1–32.
Berger, J. O. & Wolpert, R. L. (1988). The Likelihood Principle (2nd ed.). Haywood, CA: The Institute of Mathematical Statistics.
Bertsch, A., Alon, U., Neubig, G., & Gormley, M. R. (2023). Unlimiformer: Long-range transformers with unlimited length input. https://arxiv.org/abs/2305.01625
Bérut, A. et al. (2012). Experimental verification of Landauer’s principle linking information and thermodynamics. Nature, 483, 187–189. doi:10.1038/nature10872.
Bérut, A., Petrosyan, A., & Ciliberto, S. (2015). Information and thermodynamics: Experimental verification of Landauer’s erasure principle. https://arxiv.org/abs/1503.06537
Bhargava, A., Witkowski, C., Shah, M., & Thomson, M. (2023). What’s the magic word? A control theory of LLM prompting. https://arxiv.org/abs/2310.04444
Bhattiprolu, P. N., Martin, S. P., & Wells, J. D. (2020). Criteria for projected discovery and exclusion sensitivities of counting experiments. https://arxiv.org/abs/2009.07249
Billings, D., Davidson, A., Schaeffer, J., & Szafron, D. (2002). The challenge of poker. Artificial Intelligence, 134, 201–240. https://doi.org/10.1016/S0004-3702(01)00130-8
Billings, D. et al. (2003). Approximating game-theoretic optimal strategies for full-scale poker. IJCAI, 3, 661. http://webdocs.cs.ualberta.ca/~duane/publications/pdf/2003ijcai.pdf
Birnbaum, A. (1962). On the foundations of statistical inference. Journal of the American Statistical Association, 57, 269–326.
Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer.
Blondel, M., Martins, A. F., & Niculae, V. (2020). Learning with Fenchel-Young losses. Journal of Machine Learning Research, 21, 1–69.
Bogatskiy, A. et al. (2023). Explainable equivariant neural networks for particle physics: PELICAN. https://arxiv.org/abs/2307.16506
Bottou, L. (1998). Stochastic gradient descent tricks. In G. B. Orr & K. R. Muller (Eds.), Neural Networks: Tricks of the trade. Springer. https://www.microsoft.com/en-us/research/publication/stochastic-gradient-tricks/
Bousquet, O. et al. (2021). A theory of universal learning. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing (pp. 532–541). https://dl.acm.org/doi/pdf/10.1145/3406325.3451087
Bowling, M., Burch, N., Johanson, M., & Tammelin, O. (2015). Heads-up limit hold’em poker is solved. Science, 347, 145–149. http://science.sciencemag.org/content/347/6218/145
Bronstein, M. M., Bruna, J., Cohen, T., & Velickovic, P. (2021). Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. https://arxiv.org/abs/2104.13478
Brown, L. D., Cai, T. T., & DasGupta, A. (2001). Interval estimation for a binomial proportion. Statistical Science, 16, 101–133. https://projecteuclid.org/euclid.ss/1009213286
Brown, N. (2020). Equilibrium finding for large adversarial imperfect-information games. (Ph.D. thesis). http://www.cs.cmu.edu/~noamb/thesis.pdf
Brown, N., Bakhtin, A., Lerer, A., & Gong, Q. (2020). Combining deep reinforcement learning and search for imperfect-information games. https://arxiv.org/abs/2007.13544
Brown, N., Lerer, A., Gross, S., & Sandholm, T. (2019). Deep counterfactual regret minimization. https://arxiv.org/abs/1811.00164
Brown, N. & Sandholm, T. (2017). Safe and nested subgame solving for imperfect-information games. https://arxiv.org/abs/1705.02955
———. (2018). Superhuman AI for heads-up no-limit poker: Libratus beats top professionals. Science, 359, 418–424. https://science.sciencemag.org/content/359/6374/418
———. (2019a). Solving imperfect-information games via discounted regret minimization. Proceedings of the AAAI Conference on Artificial Intelligence, 33, 1829–1836. https://arxiv.org/abs/1809.04040
———. (2019b). Superhuman AI for multiplayer poker. Science, 365, 885–890. https://science.sciencemag.org/content/365/6456/885
Brown, T.B. et al. (2020). Language models are few-shot learners. (Paper on the GPT-3 model by OpenAI). https://arxiv.org/abs/2005.14165
Bubeck, S. & Sellke, M. (2023). A universal law of robustness via isoperimetry. Journal of the ACM, 70, 1–18. https://dl.acm.org/doi/full/10.1145/3578580
Bulatov, A., Kuratov, Y., & Burtsev, M. S. (2022). Recurrent memory transformer. https://arxiv.org/abs/2207.06881
———. (2023). Scaling transformer to 1M tokens and beyond with RMT. https://arxiv.org/abs/2304.11062
Burch, N. (2018). Time and Space: Why imperfect information games are hard. University of Alberta. (Ph.D. thesis). https://era.library.ualberta.ca/items/db44409f-b373-427d-be83-cace67d33c41/view/bcb00dca-39e6-4c43-9ec2-65026a50135e/Burch_Neil_E_201712_PhD.pdf
Burch, N., Lanctot, M., Szafron, D., & Gibson, R. (2012). Efficient Monte Carlo counterfactual regret minimization in games with many player actions. Advances in Neural Information Processing Systems, 25. https://proceedings.neurips.cc/paper/2012/file/3df1d4b96d8976ff5986393e8767f5b2-Paper.pdf
Burch, N., Moravcik, M., & Schmid, M. (2019). Revisiting CFR+ and alternating updates. Journal of Artificial Intelligence Research, 64, 429–443. https://www.jair.org/index.php/jair/article/view/11370
Caballero, E., Gupta, K., Rish, I., & Krueger, D. (2022). Broken neural scaling laws. https://arxiv.org/abs/2210.14891
Caldeira, J. & Nord, B. (2020). Deeply uncertain: comparing methods of uncertainty quantification in deep learning algorithms. Machine Learning: Science and Technology, 2, 015002. https://iopscience.iop.org/article/10.1088/2632-2153/aba6f3
Calin, O. & Udriste, C. (2014). Geometric Modeling in Probability and Statistics. Springer Switzerland.
Canatar, A., Bordelon, B., & Pehlevan, C. (2020). Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks. https://arxiv.org/abs/2006.13198
Cantwell, G. T. (2022). Approximate sampling and estimation of partition functions using neural networks. https://arxiv.org/abs/2209.10423
Carnap, R. (1945). The two concepts of probability. Philosophy and Phenomenological Research, 5, 513–32.
———. (1947). Probability as a guide in life. Journal of Philosophy, 44, 141–48.
———. (1952). The Continuum of Inductive Methods. University of Chicago Press. https://www.phil.cmu.edu/projects/carnap/editorial/latex_pdf/1952-1.pdf
———. (1953). What is probability? Scientific American, 189, 128–139. https://www.jstor.org/stable/24944342
Casadei, D. (2012). Estimating the selection efficiency. Journal of Instrumentation, 7, 08021. https://arxiv.org/abs/0908.0130
Cesa-Bianchi, N. & Lugosi, G. (2006). Prediction, Learning, and Games. Cambridge University Press. https://ii.uni.wroc.pl/~lukstafi/pmwiki/uploads/AGT/Prediction_Learning_and_Games.pdf
Chang, X., Li, Y., Oymak, S., & Thrampoulidis, C. (2020). Provable benefits of overparameterization in model compression: From double descent to pruning neural networks. https://arxiv.org/abs/2012.08749
Chen, R. T. Q., Rubanova, Y., Bettencourt, J., & Duvenaud, D. (2018). Neural ordinary differential equations. https://arxiv.org/abs/1806.07366
Chen, S., Dobriban, E., & Lee, J. H. (2020). A group-theoretic framework for data augmentation. https://arxiv.org/abs/1907.10905
Chen, T. & Guestrin, C. (2016). Xgboost: A scalable tree boosting system. https://arxiv.org/abs/1603.02754
Chen, X. et al. (2018). Open is not enough. Nature Physics, 15, 113–119. https://www.nature.com/articles/s41567-018-0342-2
Chevalley, M., Schwab, P., & Mehrjou, A. (2024). Deriving causal order from single-variable interventions: Guarantees & algorithm. https://arxiv.org/abs/2405.18314
Chiley, V. et al. (2019). Online normalization for training neural networks. NeurIPS 2019. https://arxiv.org/abs/1905.05894
Chowdhery, A. et al. (2022). PaLM: Scaling language modeling with pathways. https://arxiv.org/abs/2204.02311
Church, K. W. & Hestness, J. (2019). A survey of 25 years of evaluation. Natural Language Engineering, 25, 753–767. https://www.cambridge.org/core/journals/natural-language-engineering/article/survey-of-25-years-of-evaluation/E4330FAEB9202EC490218E3220DDA291
Cilibrasi, R. & Vitanyi, P. M. B. (2005). Clustering by compression. IEEE Transactions on Information Theory, 51, 1523–1545.
Ciresan, D., Meier, U., Masci, J., & Schmidhuber, J. (2012). Multi-column deep neural network for traffic sign classification. Neural Networks, 32, 333–338. https://arxiv.org/abs/1202.2745
Clopper, C. J. & Pearson, E. S. (1934). The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika, 26, 404–413.
Coadou, Y. (2022). Boosted decision trees. https://arxiv.org/abs/2206.09645
Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155–9. https://www2.psych.ubc.ca/~schaller/528Readings/Cohen1992.pdf
Cohen, T. S., Weiler, M., Kicanaoglu, B., & Welling, M. (2019). Gauge equivariant convolutional networks and the icosahedral CNN. https://arxiv.org/abs/1902.04615
Cohen, T. S. & Welling, M. (2016). Group equivariant convolutional networks. Proceedings of International Conference on Machine Learning, 2016, 2990–9. http://proceedings.mlr.press/v48/cohenc16.pdf
Collobert, R., Hannun, A., & Synnaeve, G. (2019). A fully differentiable beam search decoder. International Conference on Machine Learning, 2019, 1341–1350. http://proceedings.mlr.press/v97/collobert19a/collobert19a.pdf
Cousins, R. D. (2018). Lectures on statistics in theory: Prelude to statistics in practice. https://arxiv.org/abs/1807.05996
Cousins, R. D. & Highland, V. L. (1992). Incorporating systematic uncertainties into an upper limit. Nuclear Instruments and Methods in Physics Research Section A, 320, 331–335. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.193.1581&rep=rep1&type=pdf
Cowan, G. (1998). Statistical Data Analysis. Clarendon Press.
———. (2012). Discovery sensitivity for a counting experiment with background uncertainty. https://www.pp.rhul.ac.uk/~cowan/stat/notes/medsigNote.pdf
———. (2016). Statistics. In C. Patrignani et al. (Particle Data Group),. Chinese Physics C, 40, 100001. http://pdg.lbl.gov/2016/reviews/rpp2016-rev-statistics.pdf
Cowan, G., Cranmer, K., Gross, E., & Vitells, O. (2011). Asymptotic formulae for likelihood-based tests of new physics. European Physical Journal C, 71, 1544. https://arxiv.org/abs/1007.1727
———. (2012). Asymptotic distribution for two-sided tests with lower and upper boundaries on the parameter of interest. https://arxiv.org/abs/1210.6948
Cox, D. R. (2006). Principles of Statistical Inference. Cambridge University Press.
Cramér, H. (1946). A contribution to the theory of statistical estimation. Skandinavisk Aktuarietidskrift, 29, 85–94.
Cranmer, K. (2015). Practical statistics for the LHC. https://arxiv.org/abs/1503.07622
Cranmer, K., Brehmer, J., & Louppe, G. (2019). The frontier of simulation-based inference. https://arxiv.org/abs/1911.01429
Cranmer, K. et al. (2012). HistFactory: A tool for creating statistical models for use with RooFit and RooStats. Technical Report: CERN-OPEN-2012-016. http://inspirehep.net/record/1236448/
Cranmer, K., Pavez, J., & Louppe, G. (2015). Approximating likelihood ratios with calibrated discriminative classifiers. https://arxiv.org/abs/1506.02169
Cranmer, K., Seljak, U., & Terao, K. (2021). Machine learning. In P. A. Z. et al. (Ed.), Progress of Theoretical and Experimental Physics. 2020, 083C01. (and 2021 update). https://pdg.lbl.gov/2021-rev/2021/reviews/contents_sports.html
Cranmer, M. et al. (2020). Discovering symbolic models from deep learning with inductive biases. https://arxiv.org/abs/2006.11287
D’Agnolo, R. T. & Wulzer, A. (2019). Learning New Physics from a Machine. Physical Review D, 99, 015014. https://arxiv.org/abs/1806.02350
Dao, T. et al. (2022). FlashAttention: Fast and memory-efficient exact attention with IO-awareness. https://arxiv.org/abs/2205.14135
Dao, T. & Gu, A. (2024). Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. https://arxiv.org/abs/2405.21060
Dar, Y., Muthukumar, V., & Baraniuk, R. G. (2021). A farewell to the bias-variance tradeoff? An overview of the theory of overparameterized machine learning. https://arxiv.org/abs/2109.02355
Dawid, A. P. (2014). Discussion of "On the Birnbaum Argument for the Strong Likelihood Principle". Statistical Science, 29, 240–241. https://projecteuclid.org/journals/statistical-science/volume-29/issue-2/Discussion-of-On-the-Birnbaum-Argument-for-the-Strong-Likelihood/10.1214/14-STS470.full
de Carvalho, M., Page, G. L., & Barney, B. J. (2019). On the geometry of Bayesian inference. Bayesian Analysis, 14, 1013–1036. https://projecteuclid.org/journals/bayesian-analysis/volume-14/issue-4/On-the-Geometry-of-Bayesian-Inference/10.1214/18-BA1112.full
Denby, B. (1988). Neural networks and cellular automata in experimental high energy physics. Computer Physics Communications, 49, 429–448. https://inis.iaea.org/collection/NCLCollectionStore/_Public/20/013/20013339.pdf
———. (1993). The use of neural networks in high-energy physics. Neural Computation, 5, 505–549. https://lss.fnal.gov/archive/1992/pub/Pub-92-215-E.pdf
Dennett, D. C. (1991). Real patterns. The Journal of Philosophy, 88, 27–51. https://web.ics.purdue.edu/~drkelly/DCDRealPatterns1991.pdf
Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023). QLoRA: Efficient finetuning of quantized LLMs. https://arxiv.org/abs/2305.14314
Devlin, J., Chang, M., Lee, K., & Toutanova, K. (2018). BERT: Pre-training of deep bidirectional transformers for language understanding. https://arxiv.org/abs/1810.04805
Dhariwal, P. & Nichol, A. (2021). Diffusion models beat GANs on image synthesis. https://arxiv.org/abs/2105.05233
Dieleman, S., Fauw, J. D., & Kavukcuoglu, K. (2016). Exploiting cyclic symmetry in convolutional neural networks. https://arxiv.org/abs/1602.02660
Dinan, E., Yaida, S., & Zhang, S. (2023). Effective theory of transformers at initialization. https://arxiv.org/abs/2304.02034
Dosovitskiy, A. et al. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. https://arxiv.org/abs/2010.11929
Edelman, B. L., Goel, S., Kakade, S., & Zhang, C. (2021). Inductive biases and variable creation in self-attention mechanisms. https://arxiv.org/abs/2110.10090
Edwards, A. W. F. (1974). The history of likelihood. International Statistical Review, 42, 9–15.
Efron, B. & Hastie, T. (2016). Computer Age Statistical Inference: Algorithms, evidence, and data science. Cambridge University Press.
Elhage, N. et al. (2022). Toy models of superposition. https://transformer-circuits.pub/2022/toy_model/index.html
Evans, M. (2013). What does the proof of Birnbaum’s theorem prove? https://arxiv.org/abs/1302.5468
Fang, Z. et al. (2022). Is out-of-distribution detection learnable? NeurIPS 2022. https://arxiv.org/abs/2210.14707
Fefferman, C., Mitter, S., & Narayanan, H. (2016). Testing the manifold hypothesis. Journal of the American Mathematical Society, 29, 983–1049. https://www.ams.org/journals/jams/2016-29-04/S0894-0347-2016-00852-4/S0894-0347-2016-00852-4.pdf
Feldman, G. J. & Cousins, R. D. (1998). A unified approach to the classical statistical analysis of small signals. Physical Review D, 57, 3873. https://arxiv.org/abs/physics/9711021
Fienberg, S. E. (2006). When did Bayesian inference become "Bayesian"? Bayesian Analysis, 1, 1–40. https://projecteuclid.org/journals/bayesian-analysis/volume-1/issue-1/When-did-Bayesian-inference-become-Bayesian/10.1214/06-BA101.full
Firth, J. R. (1957). A synopsis of linguistic theory, 1930-1955. In Studies in Linguistic Analysis (pp. 1–31). Oxford: Blackwell.
Fisher, R. A. (1912). On an absolute criterion for fitting frequency curves. Statistical Science, 12, 39–41.
———. (1915). Frequency distribution of the values of the correlation coefficient in samples of indefinitely large population. Biometrika, 10, 507–521.
———. (1921). On the "probable error" of a coefficient of correlation deduced from a small sample. Metron, 1, 1–32.
———. (1935). The Design of Experiments. Hafner.
———. (1955). Statistical methods and scientific induction. Journal of the Royal Statistical Society, Series B, 17, 69–78.
Frankle, J. & Carbin, M. (2018). The lottery ticket hypothesis: Finding sparse, trainable neural networks. https://arxiv.org/abs/1803.03635
Freund, Y. & Schapire, R. E. (1997). A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences, 55, 119–139. https://doi.org/10.1006/jcss.1997.1504
Fréchet, M. (1943). Sur l’extension de certaines évaluations statistiques au cas de petits échantillons. Revue de l’Institut International de Statistique, 11, 182–205.
Fuchs, F. B., Worrall, D. E., Fischer, V., & Welling, M. (2020). SE(3)-Transformers: 3D roto-translation equivariant attention networks. https://arxiv.org/abs/2006.10503
Fukushima, K. & Miyake, S. (1982). Neocognitron: A new algorithm for pattern recognition tolerant of deformations and shifts in position. Pattern Recognition, 15, 455–469.
Gamba, M., Englesson, E., Björkman, M., & Azizpour, H. (2022). Deep double descent via smooth interpolation. https://arxiv.org/abs/2209.10080
Gandenberger, G. (2015). A new proof of the likelihood principle. British Journal for the Philosophy of Science, 66, 475–503. https://www.journals.uchicago.edu/doi/abs/10.1093/bjps/axt039
———. (2016). Why I am not a likelihoodist. Philosopher’s Imprint, 16, 1–22. https://quod.lib.umich.edu/p/phimp/3521354.0016.007/--why-i-am-not-a-likelihoodist
Gao, Y. & Chaudhari, P. (2020). An information-geometric distance on the space of tasks. https://arxiv.org/abs/2011.00613
Gelman, A. & Hennig, C. (2017). Beyond subjective and objective in statistics. Journal of the Royal Statistical Society: Series A (Statistics in Society), 180, 967–1033.
Gelman, A. & Vehtari, A. (2021). What are the most important statistical ideas of the past 50 years? Journal of the American Statistical Association, 116, 2087–2097. https://www.tandfonline.com/doi/full/10.1080/01621459.2021.1938081
Geshkovski, B., Letrouit, C., Polyanskiy, Y., & Rigollet, P. (2023). A mathematical perspective on Transformers. https://arxiv.org/abs/2312.10794
Ghosh, N. & Belkin, M. (2022). A universal trade-off between the model size, test loss, and training loss of linear predictors. https://arxiv.org/abs/2207.11621
Gibson, R. (2014). Regret minimization in games and the development of champion multiplayer computer poker-playing agents. University of Alberta. (Ph.D. thesis). https://era.library.ualberta.ca/items/15d28cbf-49d4-42e5-a9c9-fc55b1d816af/view/5ee708c7-6b8b-4b96-b1f5-23cdd95b6a46/Gibson_Richard_Spring-202014.pdf
Goldreich, O. & Ron, D. (1997). On universal learning algorithms. Information Processing Letters, 63, 131–136. https://www.wisdom.weizmann.ac.il/~oded/p_ul.html
Golovneva, O., Wang, T., Weston, J., & Sukhbaatar, S. (2024). Contextual position encoding: Learning to count what’s important. https://arxiv.org/abs/2405.18719
Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. http://www.deeplearningbook.org
Goodman, S. N. (1999a). Toward evidence-based medical statistics 1: The P value fallacy. Annals of Internal Medicine, 130, 995–1004. https://courses.botany.wisc.edu/botany_940/06EvidEvol/papers/goodman1.pdf
———. (1999b). Toward evidence-based medical statistics 2: The Bayes factor. Annals of Internal Medicine, 130, 1005–1013. https://courses.botany.wisc.edu/botany_940/06EvidEvol/papers/goodman2.pdf
Gorard, S. & Gorard, J. (2016). What to do instead of significance testing? Calculating the ’number of counterfactual cases needed to disturb a finding’. International Journal of Social Research Methodology, 19, 481–490.
Graves, A. (2013). Generating sequences with recurrent neural networks. https://arxiv.org/abs/1308.0850
Grinsztajn, L., Oyallon, E., & Varoquaux, G. (2022). Why do tree-based models still outperform deep learning on tabular data? https://arxiv.org/abs/2207.08815
Gu, A. & Dao, T. (2023). Mamba: Linear-time sequence modeling with selective state spaces. https://arxiv.org/abs/2312.00752
Gu, A., Goel, K., & Ré, C. (2021). Efficiently modeling long sequences with structured state spaces. https://arxiv.org/abs/2111.00396
Gurnee, W. et al. (2023). Finding neurons in a haystack: Case studies with sparse probing. https://arxiv.org/abs/2305.01610
Gurnee, W. & Tegmark, M. (2023). Language models represent space and time. https://arxiv.org/abs/2310.02207
Habara, K., Fukuda, E. H., & Yamashita, N. (2023). Convergence analysis and acceleration of the smoothing methods for solving extensive-form games. https://arxiv.org/abs/2303.11046
Haber, E. & Ruthotto, L. (2017). Stable architectures for deep neural networks. https://arxiv.org/abs/1705.03341
Hacking, I. (1965). Logic of Statistical Inference. Cambridge University Press.
———. (1971). Jacques Bernoulli’s Art of conjecturing. The British Journal for the Philosophy of Science, 22, 209–229.
Halverson, J., Maiti, A., & Stoner, K. (2020). Neural networks and quantum field theory. https://arxiv.org/abs/2008.08601
Hanley, J. A. & Lippman-Hand, A. (1983). If nothing goes wrong, is everything all right?: Interpreting zero numerators. JAMA, 249, 1743–1745.
Hart, S. & Mas‐Colell, A. (2000). A simple adaptive procedure leading to correlated equilibrium. Econometrica, 68, 1127–1150. https://www.ma.imperial.ac.uk/~dturaev/Hart0.pdf
Hastie, T., Montanari, A., Rosset, S., & Tibshirani, R. J. (2022). Surprises in high-dimensional ridgeless least squares interpolation. Annals of Statistics, 50, 949. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9481183/
Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2nd ed.). Springer.
He, K., Zhang, X., Ren, S., & Sun, J. (2015). Deep residual learning for image recognition. https://arxiv.org/abs/1512.03385
Heinrich, J. & Lyons, L. (2007). Systematic errors. Annual Reviews of Nuclear and Particle Science, 57, 145–169. https://www.annualreviews.org/doi/abs/10.1146/annurev.nucl.57.090506.123052
Heinrich, J. & Silver, D. (2016). Deep reinforcement learning from self-play in imperfect-information games. https://arxiv.org/abs/1603.01121
Henighan, T. et al. (2023). Superposition, memorization, and double descent. https://transformer-circuits.pub/2023/toy-double-descent/index.html
Hennig, C. (2015). What are the true clusters? Pattern Recognition Letters, 64, 53–62. https://arxiv.org/abs/1502.02555
Hestness, J. et al. (2017). Deep learning scaling is predictable, empirically. https://arxiv.org/abs/1712.00409
Hochreiter, S. & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9, 1735–1780.
Hoffmann, J. et al. (2022). Training compute-optimal large language models. https://arxiv.org/abs/2203.15556
Holzmüller, D. (2020). On the universality of the double descent peak in ridgeless regression. https://arxiv.org/abs/2010.01851
Horacek, M. (2022). Risk-Aversion in Algorithms for Poker. https://is.muni.cz/th/ydbvx/thesis.pdf
Hornik, K., Stinchcombe, M., & White, H. (1989). Multilayer feedforward networks are universal approximators. Neural Networks, 2, 359–366. https://cognitivemedium.com/magic_paper/assets/Hornik.pdf
Howard, A.G. et al. (2017). MobileNets: Efficient convolutional neural networks for mobile vision applications. https://arxiv.org/abs/1704.04861
Howard, J. N., Mandt, S., Whiteson, D., & Yang, Y. (2021). Foundations of a fast, data-driven, machine-learned simulator. https://arxiv.org/abs/2101.08944
Hu, E.J. et al. (2021). LoRA: Low-rank adaptation of large language models. https://arxiv.org/abs/2106.09685
Huang, L. et al. (2020). Normalization techniques in training DNNs: Methodology, analysis and application. https://arxiv.org/abs/2009.12836
Huh, M., Cheung, B., Wang, T., & Isola, P. (2024). The platonic representation hypothesis. https://arxiv.org/abs/2405.07987
Hutchins, J. (2000). Yehoshua Bar-Hillel: A philosophers’ contribution to machine translation.
Hutter, M. (2007). Universal Algorithmic Intelligence: A mathematical top-down approach. In Artificial General Intelligence (pp. 227–290). Springer. http://www.hutter1.net/ai/aixigentle.htm
Ingrosso, A. & Goldt, S. (2022). Data-driven emergence of convolutional structure in neural networks. https://arxiv.org/abs/2202.00565
Ioannidis, J. P. (2005). Why most published research findings are false. PLOS Medicine, 2, 696–701.
Ismael, J. (2023). Reflections on the asymmetry of causation. Interface Focus, 13, 20220081. https://royalsocietypublishing.org/doi/pdf/10.1098/rsfs.2022.0081
Ismailov, V. (2020). A three layer neural network can represent any multivariate function. https://arxiv.org/abs/2012.03016
Jamali, M. et al. (2024). Semantic encoding during language comprehension at single-cell resolution. Nature, 631, 610–616. https://www.nature.com/articles/s41586-024-07643-2
James, F. (2006). Statistical Methods in Experimental Particle Physics (2nd ed.). World Scientific.
James, F. & Roos, M. (1975). MINUIT: A system for function minimization and analysis of the parameter errors and corrections. Computational Physics Communications, 10, 343–367. https://cds.cern.ch/record/310399
James, W. & Stein, C. (1961). Estimation with quadratic loss. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Vol. 1 (pp. 361–379). University of California Press. https://projecteuclid.org/accountAjax/Download?urlId=bsmsp%2F1200512173&downloadType=presschapter&isResultClick=True
Jaynes, E. T. (2003). Probability Theory: The Logic of Science. https://bayes.wustl.edu/etj/prob/book.pdf
Jevons, W. S. (1873a). The philosophy of inductive inference. Fortnightly Review, 14, 457–476. London: Chapman and Hall.
———. (1873b). The use of hypothesis. Fortnightly Review, 14, 778–788. London: Chapman and Hall.
Jiao, L. et al. (2024). AI meets physics: A comprehensive survey. Artificial Intelligence Review, 57, 256. https://doi.org/10.1007/s10462-024-10874-4
Johanson, M. (2013). Measuring the size of large no-limit poker games. https://arxiv.org/abs/1302.7008
———. (2016). Robust Strategies and Counter-Strategies: From Superhuman to Optimal Play. University of Alberta. (Ph.D. thesis). http://johanson.ca/publications/theses/2016-johanson-phd-thesis/2016-johanson-phd-thesis.pdf
Johanson, M. et al. (2012). Efficient Nash equilibrium approximation through Monte Carlo counterfactual regret minimization. Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2012), 2, 837–846. https://www.idi.ntnu.no/emner/it3105/materials/poker/monte-carlo-cfm-2012.pdf
Johanson, M., Waugh, K., Bowling, M., & Zinkevich, M. (2011). Accelerating best response calculation in large extensive games. IJCAI 2011, Proceedings of the 22nd International Joint Conference on Artificial Intelligence, 11, 258–265. http://www.cs.cmu.edu/~kwaugh/publications/johanson11.pdf
Joyce, T. & Herrmann, J. M. (2017). A review of no free lunch theorems, and their implications for metaheuristic optimisation. In X. S. Yang (Ed.), Nature-Inspired Algorithms and Applied Optimization (pp. 27–52).
Junk, T. (1999). Confidence level computation for combining searches with small statistics. Nuclear Instruments and Methods in Physics Research Section A, 434, 435–443. https://arxiv.org/abs/hep-ex/9902006
Jurafsky, D. & Martin, J. H. (2022). Speech and Language Processing: An introduction to natural language processing, computational linguistics, and speech recognition (3rd ed.). https://web.stanford.edu/~jurafsky/slp3/ed3book_jan122022.pdf
Kaplan, J. et al. (2019). Notes on contemporary machine learning for physicists. https://sites.krieger.jhu.edu/jared-kaplan/files/2019/04/ContemporaryMLforPhysicists.pdf
———. (2020). Scaling laws for neural language models. https://arxiv.org/abs/2001.08361
Kardum, M. (2020). Rudolf Carnap–The grandfather of artificial neural networks: The influence of Carnap’s philosophy on Walter Pitts. In S. Skansi (Ed.), Guide To Deep Learning Basics: Logical, Historical And Philosophical Perspectives (pp. 55–66). Springer.
Karniadakis, G.E. et al. (2021). Physics-informed machine learning. Nature Reviews Physics, 3, 422–440. https://doi.org/10.1038/s42254-021-00314-5
Keynes, J. M. (1921). A Treatise on Probability. London: Macmillan and Co.
Kiani, B., Balestriero, R., Lecun, Y., & Lloyd, S. (2022). projUNN: efficient method for training deep networks with unitary matrices. https://arxiv.org/abs/2203.05483
Korb, K. B. (2001). Machine learning as philosophy of science. In Proceedings of the ECML-PKDD-01 Workshop on Machine Learning as Experimental Philosophy of Science. Freiburg.
Kosinski, M. (2023). Theory of mind may have spontaneously emerged in large language models. https://arxiv.org/abs/2302.02083
Kovarik, V. et al. (2022). Rethinking formal models of partially observable multiagent decision making. Artificial Intelligence, 303, 103645. https://arxiv.org/abs/1906.11110
Krenn, M. et al. (2022). On scientific understanding with artificial intelligence. Nature Reviews Physics. https://www.nature.com/articles/s42254-022-00518-3
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). ImageNet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 2012, 1097–1105. https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf
Kruschke, J. K. & Liddell, T. M. (2018). The Bayesian New Statistics: Hypothesis testing, estimation, meta-analysis, and power analysis from a Bayesian perspective. Psychonomic Bulletin & Review, 25, 178–206. https://link.springer.com/article/10.3758/s13423-016-1221-4
Kuhn, H. W. (1950). A simplified two-person poker. Contributions to the Theory of Games, 1, 97–103.
Kun, J. (2018). A Programmer’s Introduction to Mathematics. CreateSpace Independent Publishing Platform.
Lan, Z. et al. (2019). ALBERT: A lite BERT for self-supervised learning of language representations. https://arxiv.org/abs/1909.11942
Lanctot, M. (2013). Monte Carlo Sample and Regret Minimization for Equilibrium Computation and Decision-Making in Large Extensive Form Games. University of Alberta. (PhD thesis). http://mlanctot.info/files/papers/PhD_Thesis_MarcLanctot.pdf
Lanctot, M. et al. (2017). A unified game-theoretic approach to multiagent reinforcement learning. Advances in Neural Information Processing Systems, 30. https://arxiv.org/abs/1711.00832
Lanctot, M., Waugh, K., Zinkevich, M., & Bowling, M. (2009). Monte Carlo sampling for regret minimization in extensive games. Advances in Neural Information Processing Systems, 22, 1078–1086. https://proceedings.neurips.cc/paper/2009/file/00411460f7c92d2124a67ea0f4cb5f85-Paper.pdf
Lauc, D. (2020). Machine learning and the philosophical problems of induction. In S. Skansi (Ed.), Guide To Deep Learning Basics: Logical, Historical And Philosophical Perspectives (pp. 93–106). Springer.
LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521, 436–44.
LeCun, Y. & Bottou, L. (1998). Efficient BackProp. In G. B. Orr & K. R. Muller (Eds.), Neural Networks: Tricks of the trade. Springer. http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf
LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86, 2278–2324. http://vision.stanford.edu/cs598_spring07/papers/Lecun98.pdf
LeCun, Y. et al. (1989). Backpropagation applied to handwritten zip code recognition. Neural Computation, 1, 541–551. https://web.archive.org/web/20150611222615/http://yann.lecun.com/exdb/publis/pdf/lecun-89e.pdf
Leemis, L. M. & McQueston, J. T. (2008). Univariate distribution relationships. The American Statistician, 62, 45–53. http://www.stat.rice.edu/~dobelman/courses/texts/leemis.distributions.2008amstat.pdf
Lei, N., Luo, Z., Yau, S., & Gu, D. X. (2018). Geometric understanding of deep learning. https://arxiv.org/abs/1805.10451
Lewis, D. (1981). Causal decision theory. Australasian Journal of Philosophy, 59, 5–30. https://www.andrewmbailey.com/dkl/Causal_Decision_Theory.pdf
Lewis, M. et al. (2019). BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. https://arxiv.org/abs/1910.13461
Li, H. et al. (2020). Regret minimization via novel vectorized sampling policies and exploration. http://aaai-rlg.mlanctot.info/2020/papers/AAAI20-RLG_paper_14.pdf
Lin, H. & Jegelka, S. (2018). ResNet with one-neuron hidden layers is a universal approximator. https://arxiv.org/abs/1806.10909
Lista, L. (2016a). Practical statistics for particle physicists. https://arxiv.org/abs/1609.04150
———. (2016b). Statistical Methods for Data Analysis in Particle Physics. Springer. http://foswiki.oris.mephi.ru/pub/Main/Literature/st_methods_for_data_analysis_in_particle_ph.pdf
Lisy, V. & Bowling, M. (2016). Equilibrium approximation quality of current no-limit poker bots. https://arxiv.org/abs/1612.07547
Liu, H., Dai, Z., So, D. R., & Le, Q. V. (2021). Pay attention to MLPs. https://arxiv.org/abs/2105.08050
Liu, Y. et al. (2019). RoBERTa: A robustly optimized BERT pretraining approach. https://arxiv.org/abs/1907.11692
———. (2021). A survey of visual transformers. https://arxiv.org/abs/2111.06091
Liu, Z. et al. (2024). KAN: Kolmogorov-Arnold Networks. https://arxiv.org/abs/2404.19756
Liu, Z., Lin, Y., & Sun, M. (2023). Representation Learning for Natural Language Processing. Springer. https://link.springer.com/book/10.1007/978-981-99-1600-9
Liu, Z., Madhavan, V., & Tegmark, M. (2022). AI Poincare 2: Machine learning conservation laws from differential equations. https://arxiv.org/abs/2203.12610
Lovering, C. & Pavlick, E. (2022). Unit testing for concepts in neural networks. Transactions of the Association for Computational Linguistics, 10, 1193–1208. https://aclanthology.org/2022.tacl-1.69/
Lu, C. et al. (2024). The AI Scientist: Towards fully automated open-ended scientific discovery. https://arxiv.org/abs/2408.06292
Lu, Z. et al. (2017). The expressive power of neural networks: A view from the width. Advances in Neural Information Processing Systems, 30. https://proceedings.neurips.cc/paper/2017/file/32cbf687880eb1674a07bf717761dd3a-Paper.pdf
Lundberg, I., Johnson, R., & Stewart, B. M. (2021). What is your estimand? Defining the target quantity connects statistical evidence to theory. American Sociological Review, 86, 532–565. https://journals.sagepub.com/doi/abs/10.1177/00031224211004187
Lyons, L. (2008). Open statistical issues in particle physics. The Annals of Applied Statistics, 2, 887–915. https://projecteuclid.org/journals/annals-of-applied-statistics/volume-2/issue-3/Open-statistical-issues-in-Particle-Physics/10.1214/08-AOAS163.full
Ma, S. et al. (2024). The era of 1-bit LLMs: All large language models are in 1.
Ma, X. et al. (2024). Megalodon: Efficient LLM pretraining and inference with unlimited context length. https://arxiv.org/abs/2404.08801
MacKay, D. J. C. (2003). Information Theory, Inference, and Learning Algorithms. Cambridge University Press.
Maddox, W. J., Benton, G., & Wilson, A. G. (2023). Rethinking parameter counting in deep models: Effective dimensionality revisited. https://arxiv.org/abs/2003.02139
Mahowald, K. et al. (2023). Dissociating language and thought in large language models: a cognitive perspective. https://arxiv.org/abs/2301.06627
Marchetti, G. L., Hillar, C., Kragic, D., & Sanborn, S. (2023). Harmonics of learning: Universal fourier features emerge in invariant networks. https://arxiv.org/abs/2312.08550
Mayo, D. G. (1981). In defense of the Neyman-Pearson theory of confidence intervals. Philosophy of Science, 48, 269–280.
———. (1996). Error and the Growth of Experimental Knowledge. Chicago University Press.
———. (2014). On the Birnbaum Argument for the Strong Likelihood Principle,. Statistical Science, 29, 227–266.
———. (2018). Statistical Inference as Severe Testing: How to Get Beyond the Statistics Wars. Cambridge University Press.
———. (2019). The law of likelihood and error statistics. https://errorstatistics.com/2019/04/04/excursion-1-tour-ii-error-probing-tools-versus-logics-of-evidence-excerpt/
———. (2021). Significance tests: Vitiated or vindicated by the replication crisis in psychology? Review of Philosophy and Psychology, 12, 101–121. https://link.springer.com/article/10.1007/s13164-020-00501-w
Mayo, D. G. & Spanos, A. (2006). Severe testing as a basic concept in a Neyman-Pearson philosophy of induction. British Journal for the Philosophy of Science, 57, 323–357.
———. (2011). Error statistics. In Philosophy of Statistics (pp. 153–198). North-Holland.
McCarthy, J., Minsky, M. L., Rochester, N., & Shannon, C. E. (1955). A proposal for the Dartmouth Summer Research Project on Artificial Intelligence. http://www-formal.stanford.edu/jmc/history/dartmouth.pdf
McDermott, J. (2019). When and why metaheuristics researchers can ignore "no free lunch" theorems. https://arxiv.org/abs/1906.03280
McDougall, C. et al. (2023). Copy suppression: Comprehensively understanding an attention head. https://arxiv.org/abs/2310.04625
McFadden, D. & Zarembka, P. (1973). Conditional logit analysis of qualitative choice behavior. In Frontiers in Econometrics (pp. 105–142). New York: Academic Press.
Meehl, P. E. (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology, 46, 806–834.
Meng, K., Bau, D., Andonian, A., & Belinkov, Y. (2023). Locating and editing factual associations in GPT https://arxiv.
Merrill, W. & Sabharwal, A. (2022). The parallelism tradeoff: Limitations of log-precision transformers. https://arxiv.org/abs/2207.00729
Mialon, G. et al. (2023). Augmented Language Models: a Survey. https://arxiv.org/abs/2302.07842
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. https://arxiv.org/abs/1301.3781
Mikolov, T. et al. (2013). Distributed representations of words and phrases and their compositionality. https://arxiv.org/abs/1310.4546
Mikolov, T., Yih, W. T., & Zweig, G. (2013). Linguistic regularities in continuous space word representations. NAACL HLT 2013. https://www.aclweb.org/anthology/N13-1090.pdf
Minsky, M. & Papert, S. (1969). Perceptrons: An Introduction to Computational Geometry. MIT Press.
Mitchell, T. M. (1980). The need for biases in learning generalizations. In Readings in Machine Learning (pp. 184–192). San Mateo, CA, USA. http://www.cs.cmu.edu/afs/cs/usr/mitchell/ftp/pubs/NeedForBias_1980.pdf
Mnih, V. et al. (2013). Playing Atari with deep reinforcement learning. https://arxiv.org/abs/1312.5602
———. (2015). Human-level control through deep reinforcement learning. Nature, 518, 529–533. http://files.davidqiu.com//research/nature14236.pdf
Mohamadi, S. et al. (2023). ChatGPT in the age of generative AI and large language models: A concise survey. https://arxiv.org/abs/2307.04251v1
Moravcik, M. et al. (2017). DeepStack: Expert-level artificial intelligence in heads-up no-limit poker. Science, 356, 508–513. https://arxiv.org/abs/1701.01724
Muennighoff, N. et al. (2023). Scaling data-constrained language models. https://arxiv.org/abs/2305.16264
Murphy, K. P. (2012). Machine Learning: A probabilistic perspective. MIT Press.
———. (2022). Probabilistic Machine Learning: An introduction. MIT Press.
Muthukumar, V., Vodrahalli, K., Subramanian, V., & Sahai, A. (2019). Harmless interpolation of noisy data in regression. https://arxiv.org/abs/1903.09139
Nagarajan, V. (2021). Explaining generalization in deep learning: progress and fundamental limits. (Ph.D. thesis). https://arxiv.org/abs/2110.08922
Nakkiran, P. (2021). Turing-universal learners with optimal scaling laws. https://arxiv.org/abs/2111.05321
Nakkiran, P., Bradley, A., Zhou, H., & Advani, M. (2024). Step-by-step diffusion: An elementary tutorial. https://arxiv.org/abs/2406.08929
Nakkiran, P. et al. (2019). Deep double descent: Where bigger models and more data hurt. https://arxiv.org/abs/1912.02292
Neller, T. W. & Lanctot, M. (2013). An introduction to counterfactual regret minimization. Proceedings of Model AI Assignments, 11. http://cs.gettysburg.edu/~tneller/modelai/2013/cfr/cfr.pdf
Neyman, J. (1955). The problem of inductive inference. Communications on Pure and Applied Mathematics, 8, 13–45. https://errorstatistics.files.wordpress.com/2017/04/neyman-1955-the-problem-of-inductive-inference-searchable.pdf
———. (1977). Frequentist probability and frequentist statistics. Synthese, 36, 97–131.
Neyman, J. & Pearson, E. S. (1933). On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society A, 231, 289–337.
Nielsen, F. (2013). Cramer-Rao lower bound and information geometry. https://arxiv.org/abs/1301.3578
———. (2020). An elementary introduction to information geometry. Entropy, 22, 1100. https://www.mdpi.com/1099-4300/22/10/1100
Nirenburg, S. (1996). Bar Hillel and Machine Translation: Then and Now.
Nissim, M., Noord, R. van, & Goot, R. van der. (2019). Fair is better than sensational: Man is to doctor as woman is to doctor. Computational Linguistics, 46, 487–497.
Norvig, P. (2011). On Chomsky and the Two Cultures of Statistical Learning. https://norvig.com/chomsky.html
O’Hagan, A. (2010). Kendall’s Advanced Theory of Statistics, Vol 2B: Bayesian Inference. Wiley.
OpenAI. (2023). GPT-4 Technical Report. https://cdn.openai.com/papers/gpt-4.pdf
Opper, M. (2001). Learning to generalize. Frontiers of Life, 3, 763–775.
Opper, M. & Kinzel, W. (1996). Statistical mechanics of generalization. In Models of Neural Networks III: Association, Generalization, and Representation (pp. 151–209). Springer New York. https://gwern.net/doc/ai/nn/1996-opper.pdf
Otsuka, J. (2023). Thinking About Statistics: The Philosophical Foundations. Routledge.
Ouyang, L. et al. (2022). Training language models to follow instructions with human feedback. https://arxiv.org/abs/2203.02155
Pandey, R. (2024). gzip predicts data-dependent scaling laws. https://arxiv.org/abs/2405.16684
Park, N. & Kim, S. (2022). How do vision transformers work? https://arxiv.org/abs/2202.06709
Patel, R. & Pavlick, E. (2022). Mapping language models to grounded conceptual spaces. International Conference on Learning Representations, 2022. https://openreview.net/pdf?id=gJcEM8sxHK
Pearl, J. (2009). Causal inference in statistics: An overview. Statistics Surveys, 3, 96–146. https://projecteuclid.org/journals/statistics-surveys/volume-3/issue-none/Causal-inference-in-statistics-An-overview/10.1214/09-SS057.pdf
———. (2018). The Book of Why: The new science of cause and effect. Basic Books.
Pearson, K. (1900). On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 50, 157–175.
Peirce, C. S. (1883). Studies in Logic. Boston: Little, Brown, and Co.
Peng, B. et al. (2023). RWKV: Reinventing RNNs for the Transformer Era. https://arxiv.org/abs/2305.13048
Perone, C. S. (2018). NLP word representations and the Wittgenstein philosophy of language. http://blog.christianperone.com/2018/05/nlp-word-representations-and-the-wittgenstein-philosophy-of-language/
Peters, J., Janzing, D., & Scholkopf, B. (2017). Elements of Causal Inference. MIT Press.
Phuong, M. & Hutter, M. (2022). Formal algorithms for transformers. https://arxiv.org/abs/2207.09238
Piantadosi, S. T. (2023). Modern language models refute Chomsky’s approach to language. https://lingbuzz.net/lingbuzz/007180
Ponsen, M., De Jong, S., & Lanctot, M. (2011). Computing approximate Nash equilibria and robust best-responses using sampling. Journal of Artificial Intelligence Research, 42, 575–605. https://arxiv.org/abs/1401.4591
Radford, A. et al. (2019). Language models are unsupervised multitask learners. (Paper on the GPT-2 model by OpenAI). https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf
Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving language understanding by generative pre-training. (Paper on the GPT model by OpenAI). https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf
Rae, J.W. et al. (2022). Scaling language models: Methods, analysis & insights from training Gopher. https://arxiv.org/abs/2112.11446
Raffel, C. et al. (2019). Exploring the limits of transfer learning with a unified text-to-text transformer. https://arxiv.org/abs/1910.10683
Raissi, M., Perdikaris, P., & Karniadakis, G. E. (2017a). Physics informed deep learning (Part I): Data-driven solutions of nonlinear partial differential equations. https://arxiv.org/abs/1711.10561
———. (2017b). Physics informed deep learning (Part II): Data-driven discovery of nonlinear partial differential equations. https://arxiv.org/abs/1711.10566
Rao, C. R. (1945). Information and the accuracy attainable in the estimation of statistical parameters. Bulletin of the Calcutta Mathematical Society, 37, 81–91.
———. (1947). Minimum variance and the estimation of several parameters. In Mathematical Proceedings of the Cambridge Philosophical Society. 43, 280–283. Cambridge University Press.
———. (1997). Statisitcs and Truth: Putting Chance to Work (2nd ed.). World Scientific.
Rao, C. R. & Lovric, M. M. (2016). Testing point null hypothesis of a normal mean and the truth: 21st century perspective. Journal of Modern Applied Statistical Methods, 15, 2–21. http://digitalcommons.wayne.edu/jmasm/vol15/iss2/3
Rathmanner, S. & Hutter, M. (2011). A philosophical treatise of universal induction. Entropy, 13, 1076–1136. https://www.mdpi.com/1099-4300/13/6/1076
Read, A. L. (2002). Presentation of search results: the CLs technique. Journal of Physics G: Nuclear and Particle Physics, 28, 2693. https://indico.cern.ch/event/398949/attachments/799330/1095613/The_CLs_Technique.pdf
Reid, C. (1998). Neyman. Springer-Verlag.
Rice, J. A. (2007). Mathematical Statistics and Data Analysis (3rd ed.). Thomson.
Roberts, D. A. (2021). Why is AI hard and physics simple? https://arxiv.org/abs/2104.00008
Roberts, D. A., Yaida, S., & Hanin, B. (2021). The Principles of Deep Learning Theory: An Effective Theory Approach to Understanding Neural Networks. Cambridge University Press. https://deeplearningtheory.com/PDLT.pdf
Robins, J. M. & Wasserman, L. (1999). On the impossibility of inferring causation from association without background knowledge. In C. Glymour & G. Cooper (Eds.), Computation, Causation, and Discovery (pp. 305–321). AAAI & MIT Press.
Ronen, M., Finder, S. E., & Freifeld, O. (2022). DeepDPM: Deep clustering with an unknown number of clusters. https://arxiv.org/abs/2203.14309
Rosenblatt, F. (1961). Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms. Spartan.
Roughgarden, T. (2016). Twenty Lectures on Algorithmic Game Theory. Cambridge University Press.
Royall, R. (1997). Statistical Evidence: A likelihood paradigm. CRC Press.
Rozeboom, W. W. (1960). The fallacy of the null-hypothesis significance test. Psychological Bulletin, 57, 416.
Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66, 688. https://psycnet.apa.org/fulltext/1975-06502-001.pdf
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323, 533–536. https://www.nature.com/articles/323533a0.pdf
Salsburg, D. (2001). The Lady Tasting Tea. Holt.
Savage, L. J. (1954). The Foundations of Statistics. John Wiley & Sons.
Scardapane, S. (2024). Alice’s Adventures in a Differentiable Wonderland, Vol. I: A Tour of the Land. https://arxiv.org/abs/2404.17625
Schaeffer, R. et al. (2023). Double descent demystified: Identifying, interpreting & ablating the sources of a deep learning puzzle. https://arxiv.org/abs/2303.14151
Schmid, M. et al. (2019). Variance reduction in Monte Carlo counterfactual regret minimization (VR-MCCFR) for extensive form games using baselines. https://ojs.aaai.org/index.php/AAAI/article/view/4048/3926
———. (2021). Player of games. https://arxiv.org/abs/2112.03178
Shalev-Shwarz, S. & Ben-David, S. (2014). Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press. https://www.cs.huji.ac.il/w~shais/UnderstandingMachineLearning/understanding-machine-learning-theory-algorithms.pdf
Silver, D. et al. (2016). Mastering the game of Go with deep neural networks and tree search. Nature, 529, 484–489.
———. (2017a). Mastering chess and shogi by self-play with a general reinforcement learning algorithm. https://arxiv.org/abs/1712.01815
———. (2017b). Mastering the game of Go without human knowledge. Nature, 550, 354–359.
Silver, D., Singh, S., Precup, D., & Sutton, R. S. (2024). Reward is enough. Artificial Intelligence, 299, 103535. https://www.sciencedirect.com/science/article/pii/S0004370221000862
Simonyan, K. & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. https://arxiv.org/abs/1409.1556
Sinervo, P. (2002). Signal significance in particle physics. In M. R. Whalley & L. Lyons (Eds.), Proceedings of the Conference on Advanced Statistical Techniques in Particle Physics. Durham, UK: Institute of Particle Physics Phenomenology. https://arxiv.org/abs/hep-ex/0208005v1
———. (2003). Definition and treatment of systematic uncertainties in high energy physics and astrophysics. In Lyons L., Mount R., & R. Reitmeyer (Eds.), Proceedings of the Conference on Statistical Problems in Particle Physics, Astrophysics, and Cosmology (PhyStat2003) (pp. 122–129). Stanford Linear Accelerator Center. https://www.slac.stanford.edu/econf/C030908/papers/TUAT004.pdf
Singh, S. P., Lucchi, A., Hofmann, T., & Schölkopf, B. (2022). Phenomenology of double descent in finite-width neural networks. https://arxiv.org/abs/2203.07337
Skelac, I. & Jandric, A. (2020). Meaning as use: From Wittgenstein to Google’s Word2vec. In S. Skansi (Ed.), Guide To Deep Learning Basics: Logical, Historical And Philosophical Perspectives (pp. 41–53). Springer.
Slonim, N., Atwal, G. S., Tkacik, G., & Bialek, W. (2005). Information-based clustering. Proceedings of the National Academy of Sciences, 102, 18297–18302. https://arxiv.org/abs/q-bio/0511043
Smith, L. (2019). A gentle introduction to information geometry. September 27, 2019. http://www.robots.ox.ac.uk/~lsgs/posts/2019-09-27-info-geom.html
Sohl-Dickstein, J. (2020). Two equalities expressing the determinant of a matrix in terms of expectations over matrix-vector products. https://arxiv.org/abs/2005.06553
Solomonoff, G. (2016). Ray Solomonoff and the Dartmouth Summer Research Project in Artificial Intelligence, 1956. http://raysolomonoff.com/dartmouth/dartray.pdf
Southey, F. et al. (2012). Bayes’ bluff: Opponent modelling in poker. https://arxiv.org/abs/1207.1411
Spears, B.K. et al. (2018). Deep learning: A guide for practitioners in the physical sciences. Physics of Plasmas, 25, 080901.
Stahlberg, F. (2019). Neural machine translation: A review. https://arxiv.org/abs/1912.02047
Stein, C. (1956). Inadmissibility of the usual estimator for the mean of a multivariate normal distribution. Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability, 1, 197–206.
Steinhardt, J. (2012). Beyond Bayesians and frequentists. https://jsteinhardt.stat.berkeley.edu/files/stats-essay.pdf
———. (2022). More is different for AI. https://bounded-regret.ghost.io/more-is-different-for-ai/
Stuart, A., Ord, K., & Arnold, S. (2010). Kendall’s Advanced Theory of Statistics, Vol 2A: Classical Inference and the Linear Model. Wiley.
Sun, Y. et al. (2023). Retentive network: A successor to transformer for large language models. https://arxiv.org/abs/2307.08621
Sutskever, I. (2015). A brief overview of deep learning. https://web.archive.org/web/20220728224752/http://yyue.blogspot.com/2015/01/a-brief-overview-of-deep-learning.html
Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to sequence learning with neural networks. Advances in Neural Information Processing Systems, 2014, 3104–3112. https://arxiv.org/abs/1409.3215
Sutton, R. S. (2019). The bitter lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html
Sutton, R. S. & Barto, A. G. (2018). Reinforcement Learning (2nd ed.). MIT Press.
Sznajder, M. (2018). Inductive logic as explication: The evolution of Carnap’s notion of logical probability. The Monist, 101, 417–440.
Tammelin, O. (2014). Solving large imperfect information games using CFR+. https://arxiv.org/abs/1407.5042
Tammelin, O., Burch, N., Johanson, M., & Bowling, M. (2015). Solving heads-up limit texas hold’em. International Joint Conference on Artificial Intelligence, 24. http://johanson.ca/publications/poker/2015-ijcai-cfrplus/2015-ijcai-cfrplus.pdf
Tan, M. & Le, Q. V. (2019). EfficientNet: Rethinking model scaling for convolutional neural networks. https://arxiv.org/abs/1905.11946
———. (2021). EfficientNetV2: Smaller models and faster training. https://arxiv.org/abs/2104.00298
Tay, Y., Dehghani, M., Bahri, D., & Metzler, D. (2022). Efficient transformers: A survey. https://arxiv.org/abs/2009.06732
Tegmark, M., Taylor, A. N., & Heavens, A. F. (1997). Karhunen-Loeve eigenvalue problems in cosmology: How should we tackle large data sets? The Astrophysical Journal, 480, 22–35. https://arxiv.org/abs/astro-ph/9603021
Tenney, I. et al. (2019). What do you learn from context? Probing for sentence structure in contextualized word representations. https://arxiv.org/abs/1905.06316
Thuerey, N. et al. (2021). Physics-based deep learning. https://arxiv.org/abs/2109.05237
Timbers, F. (2020). Approximate exploitability: Learning a best response in large games. https://arxiv.org/abs/2004.09677
Tukey, J. W. (1977). Exploratory Data Analysis. Pearson.
Udrescu, S. & Tegmark, M. (2020). Symbolic pregression: Discovering physical laws from raw distorted video. https://arxiv.org/abs/2005.11212
van Handel, R. (2016). Probability in high dimensions. Lecture notes at Princeton. https://web.math.princeton.edu/~rvan/APC550.pdf
Vapnik, V., Levin, E., & LeCun, Y. (1994). Measuring the VC-dimension of a learning machine. Neural Computation, 6, 851–876.
Vaswani, A. et al. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 2017, 5998–6008. https://arxiv.org/abs/1706.03762
Venn, J. (1888). The Logic of Chance. London: MacMillan and Co. (Originally published in 1866).
Vershynin, R. (2018). High-Dimensional Probability: An Introduction with Applications in Data Science. Cambridge University Press. https://www.math.uci.edu/~rvershyn/papers/HDP-book/HDP-book.pdf
Wainer, H. (2007). The most dangerous equation. American Scientist, 95, 249–256. https://sites.stat.washington.edu/people/peter/498.Sp16/Equation.pdf
Wakefield, J. (2013). Bayesian and Frequentist Regression Methods. Springer.
Wald, A. (1943). Tests of statistical hypotheses concerning several parameters when the number of observations is large. Transactions of the American Mathematical Society, 54, 426–482. https://www.ams.org/journals/tran/1943-054-03/S0002-9947-1943-0012401-3/S0002-9947-1943-0012401-3.pdf
Wang, H. et al. (2023). BitNet: Scaling 1-bit transformers for large language models. https://arxiv.org/abs/2310.11453
Wasserman, L. (2003). All of Statistics: A Concise Course in Statistical Inference. Springer.
Wasserstein, R. L., Allen, L. S., & Lazar, N. A. (2019). Moving to a World Beyond "p<0.05". American Statistician, 73, 1–19.
Wasserstein, R. L. & Lazar, N. A. (2016). The ASA’s statement on p-values: Context, process, and purpose. American Statistician, 70, 129–133.
Watson, D. & Floridi, L. (2019). The explanation game: A formal framework for interpretable machine learning. SSRN, 3509737. https://ssrn.com/abstract=3509737
Weisberg, J. (2019). Odds & Ends: Introducing Probability & Decision with a Visual Emphasis. https://jonathanweisberg.org/vip/
Werbos, P. J. (1990). Backpropagation through time: what it does and how to do it. Proceedings of the IEEE, 78, 1550–1560. http://www.werbos.com/Neural/BTT.pdf
Wilks, S. S. (1938). The large-sample distribution of the likelihood ratio for testing composite hypotheses. The Annals of Mathematical Statistics, 9, 60–62. https://projecteuclid.org/journals/annals-of-mathematical-statistics/volume-9/issue-1/The-Large-Sample-Distribution-of-the-Likelihood-Ratio-for-Testing/10.1214/aoms/1177732360.full
Williamson, J. (2009). The philosophy of science and its relation to machine learning. In Scientific Data Mining and Knowledge Discovery (pp. 77–89). Springer, Berlin, Heidelberg.
Wittgenstein, L. (2009). Philosophical Investigations. (E. Anscombe & P. Hacker, Trans., P. Hacker & J. Schulte, Eds.) (4th ed.). Wiley-Blackwell. (Originally published in 1953).
Wolfram, S. (2023). What is ChatGPT doing—and why does it work? https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/
Wolpert, D. H. (1996). The lack of a priori distinctions between learning algorithms. Neural Computation, 8, 1341–1390.
———. (2007). Physical limits of inference. https://arxiv.org/abs/0708.1362
Wolpert, D. H. & Kinney, D. (2020). Noisy deductive reasoning: How humans construct math, and how math constructs universes. https://arxiv.org/abs/2012.08298
Wolpert, D. H. & Macready, W. G. (1995). No free lunch theorems for search. Technical Report SFI-TR-95-02-010, Santa Fe Institute.
———. (1997). No free lunch theorems for optimization. IEEE Transactions on Evolutionary Computation, 1, 67–82.
Wu, Y. et al. (2016). Google’s neural machine translation system: Bridging the gap between human and machine translation. https://arxiv.org/abs/1409.0473
Yang, T. & Suzuki, J. (2023). Dropout drops double descent. https://arxiv.org/abs/2305.16179
Yang, Z. et al. (2019). XLNet: Generalized autoregressive pretraining for language understanding. https://arxiv.org/abs/1906.08237
Zaheer, M. et al. (2020). Big Bird: Transformers for longer sequences. https://arxiv.org/abs/2007.14062
Zech, G. (1995). Comparing statistical data to Monte Carlo simulation: Parameter fitting and unfolding. (DESY-95-113). Deutsches Elektronen-Synchrotron (DESY). https://cds.cern.ch/record/284321
Zhang, W. (1998). Complete anytime beam search. AAAI Proceedings, 98, 425–430. https://cdn.aaai.org/AAAI/1998/AAAI98-060.pdf
Zhao, J. et al. (2024). GaLore: Memory-efficient LLM training by gradient low-rank projection. https://arxiv.org/abs/2403.03507
Zhao, W.X. et al. (2023). A survey of large language models. https://arxiv.org/abs/2303.18223
Zhou, R. & Hansen, E. A. (2005). Beam-stack search: Integrating backtracking with beam search. ICAPS, 15, 90–98. https://cdn.aaai.org/ICAPS/2005/ICAPS05-010.pdf
Zinkevich, M., Johanson, M., Bowling, M., & Piccione, C. (2007). Regret minimization in games with incomplete information. Advances in Neural Information Processing Systems, 20, 1729–1736. https://proceedings.neurips.cc/paper/2007/file/08d98638c6fcd194a4b1e6992063e944-Paper.pdf

  1. Rao (1997), p. x.↩︎

  2. Edwards (1974), p. 9.↩︎

  3. Hacking (1971).↩︎

  4. Bernoulli, J. (1713). Ars Conjectandi, Chapter II, Part IV, defining the art of conjecture [wikiquote].↩︎

  5. Venn (1888).↩︎

  6. Jevons (1873a).↩︎

  7. Jevons (1873b).↩︎

  8. Peirce (1883), p. 126–181.↩︎

  9. Pearson (1900).↩︎

  10. Keynes (1921).↩︎

  11. Fisher (1912).↩︎

  12. Fisher (1915).↩︎

  13. Fisher (1921).↩︎

  14. Fisher (1955).↩︎

  15. Salsburg (2001).↩︎

  16. Reid (1998).↩︎

  17. Neyman (1955).↩︎

  18. Stuart, Ord, & Arnold (2010).↩︎

  19. F. James (2006).↩︎

  20. Cowan (1998) and Cowan (2016).↩︎

  21. Cranmer (2015).↩︎

  22. Jaynes (2003).↩︎

  23. Lista (2016b).↩︎

  24. Lista (2016a).↩︎

  25. Cox (2006).↩︎

  26. Behnke, Kröninger, Schott, & Schörner-Sadenius (2013).↩︎

  27. Cousins (2018).↩︎

  28. Weisberg (2019).↩︎

  29. Gelman & Vehtari (2021).↩︎

  30. Otsuka (2023).↩︎

  31. Carnap (1947).↩︎

  32. Carnap (1953).↩︎

  33. Goodfellow, Bengio, & Courville (2016), p. 72-73.↩︎

  34. Cowan (1998), p. 20-22.↩︎

  35. Arras (1998).↩︎

  36. Fienberg (2006).↩︎

  37. Weisberg (2019), ch. 15.↩︎

  38. Fisher (1921), p. 15.↩︎

  39. Stein (1956).↩︎

  40. W. James & Stein (1961).↩︎

  41. van Handel (2016).↩︎

  42. Vershynin (2018).↩︎

  43. Leemis & McQueston (2008).↩︎

  44. Cranmer, K. et al. (2012).↩︎

  45. This assumption that the model models the data “reasonably” well reflects that to the degree required by your analysis, the important features of the data match well within the systematic uncertainties parametrized within the model. If the model is incomplete because it is missing an important feature of the data, then this is the “ugly” (class-3) error in the Sinervo classification of systematic uncertainties.↩︎

  46. Cowan (1998) and Cowan (2016), p. TODO.↩︎

  47. Aldrich (1997).↩︎

  48. F. James (2006), p. 234.↩︎

  49. Cox (2006), p. 11.↩︎

  50. Murphy (2012), p. 222.↩︎

  51. Fréchet (1943), Cramér (1946), Rao (1945), and Rao (1947).↩︎

  52. Rice (2007), p. 300–2.↩︎

  53. Nielsen (2013).↩︎

  54. Cowan (1998), p. 130-5.↩︎

  55. F. James (2006), p. 234.↩︎

  56. F. James & Roos (1975).↩︎

  57. Cowan, Cranmer, Gross, & Vitells (2012).↩︎

  58. Wainer (2007).↩︎

  59. Tegmark, Taylor, & Heavens (1997).↩︎

  60. Clopper & Pearson (1934).↩︎

  61. Agresti & Coull (1998).↩︎

  62. Hanley & Lippman-Hand (1983).↩︎

  63. L. D. Brown, Cai, & DasGupta (2001).↩︎

  64. Casadei (2012).↩︎

  65. Fisher (1935), p. 16.↩︎

  66. Goodman (1999a). p. 998.↩︎

  67. J. Cohen (1992).↩︎

  68. ATLAS and CMS Collaborations (2011).↩︎

  69. Cowan, Cranmer, Gross, & Vitells (2011).↩︎

  70. Neyman & Pearson (1933).↩︎

  71. Feldman & Cousins (1998).↩︎

  72. Sinervo (2002) and Cowan (2012).↩︎

  73. Cowan et al. (2011), p. 2–3.↩︎

  74. Cowan et al. (2011), p. 3.↩︎

  75. Cousins & Highland (1992).↩︎

  76. Junk (1999).↩︎

  77. Read (2002).↩︎

  78. ATLAS Statistics Forum (2011).↩︎

  79. Wilks (1938).↩︎

  80. Wald (1943).↩︎

  81. Cowan et al. (2011).↩︎

  82. Bhattiprolu, Martin, & Wells (2020).↩︎

  83. Murphy (2012), p. 197.↩︎

  84. Goodman (1999b).↩︎

  85. Goodman (1999a). p. 995.↩︎

  86. Sinervo (2003).↩︎

  87. Heinrich & Lyons (2007).↩︎

  88. Caldeira & Nord (2020).↩︎

  89. Lyons (2008), p. 890.↩︎

  90. Rubin (1974).↩︎

  91. Lewis (1981).↩︎

  92. Pearl (2018).↩︎

  93. Pearl (2009).↩︎

  94. Robins & Wasserman (1999).↩︎

  95. Peters, Janzing, & Scholkopf (2017).↩︎

  96. Lundberg, Johnson, & Stewart (2021).↩︎

  97. Ismael (2023).↩︎

  98. Chevalley, Schwab, & Mehrjou (2024).↩︎

  99. Tukey (1977).↩︎

  100. Chen, X. et al. (2018).↩︎

  101. Carnap (1945).↩︎

  102. Royall (1997), p. 171–2.↩︎

  103. Cranmer (2015), p. 6.↩︎

  104. Neyman & Pearson (1933).↩︎

  105. Kruschke & Liddell (2018).↩︎

  106. Edwards (1974).↩︎

  107. Birnbaum (1962).↩︎

  108. Hacking (1965).↩︎

  109. Berger & Wolpert (1988).↩︎

  110. O’Hagan (2010), p. 17–18.↩︎

  111. Gandenberger (2015).↩︎

  112. Evans (2013).↩︎

  113. Mayo (2014).↩︎

  114. Mayo (2019).↩︎

  115. Dawid (2014).↩︎

  116. Mayo (2019).↩︎

  117. Lyons (2008), p. 891.↩︎

  118. Sznajder (2018).↩︎

  119. Carnap (1952).↩︎

  120. Hacking (1965).↩︎

  121. Neyman (1977).↩︎

  122. Rozeboom (1960).↩︎

  123. Meehl (1978).↩︎

  124. Zech (1995).↩︎

  125. Royall (1997).↩︎

  126. Berger (2003).↩︎

  127. Mayo (1981).↩︎

  128. Mayo (1996).↩︎

  129. Mayo & Spanos (2006).↩︎

  130. Mayo & Spanos (2011).↩︎

  131. Mayo (2018).↩︎

  132. Gelman & Hennig (2017).↩︎

  133. Murphy (2012), ch. 6.6.↩︎

  134. Murphy (2022), p. 195–198.↩︎

  135. Gandenberger (2016).↩︎

  136. Wakefield (2013), ch. 4.↩︎

  137. Efron & Hastie (2016), p. 30–36.↩︎

  138. Kruschke & Liddell (2018).↩︎

  139. Steinhardt (2012).↩︎

  140. Goodman (1999a). p. 999.↩︎

  141. Ioannidis (2005).↩︎

  142. Wasserstein & Lazar (2016).↩︎

  143. Wasserstein, Allen, & Lazar (2019).↩︎

  144. Benjamin, D.J. et al. (2017).↩︎

  145. Fisher (1935), p. 13–14.↩︎

  146. Rao & Lovric (2016).↩︎

  147. Mayo (2021).↩︎

  148. Gorard & Gorard (2016).↩︎

  149. Benjamini, Y. et al. (2021), p. 1.↩︎

  150. Hastie, Tibshirani, & Friedman (2009).↩︎

  151. MacKay (2003).↩︎

  152. Murphy (2012).↩︎

  153. Murphy (2022), p. 195–198.↩︎

  154. Shalev-Shwarz & Ben-David (2014).↩︎

  155. Vapnik, Levin, & LeCun (1994).↩︎

  156. Shalev-Shwarz & Ben-David (2014), p. 67–82.↩︎

  157. McCarthy, Minsky, Rochester, & Shannon (1955).↩︎

  158. Solomonoff (2016).↩︎

  159. Kardum (2020).↩︎

  160. Rosenblatt (1961).↩︎

  161. Minsky & Papert (1969).↩︎

  162. Murphy (2012), p. 21.↩︎

  163. Note: Label smoothing is a regularization technique that smears the activation over other labels, but we don’t do that here.↩︎

  164. “Logit” was coined by Joseph Berkson (1899-1982).↩︎

  165. McFadden & Zarembka (1973).↩︎

  166. Blondel, Martins, & Niculae (2020).↩︎

  167. Goodfellow et al. (2016), p. 129.↩︎

  168. Freund & Schapire (1997).↩︎

  169. T. Chen & Guestrin (2016).↩︎

  170. Aytekin (2022).↩︎

  171. Grinsztajn, Oyallon, & Varoquaux (2022).↩︎

  172. Coadou (2022).↩︎

  173. Slonim, Atwal, Tkacik, & Bialek (2005).↩︎

  174. Batson, Haaf, Kahn, & Roberts (2021).↩︎

  175. Hennig (2015).↩︎

  176. Lauc (2020), p. 103–4.↩︎

  177. Ronen, Finder, & Freifeld (2022).↩︎

  178. Fang, Z. et al. (2022).↩︎

  179. Bengio (2009).↩︎

  180. LeCun, Bengio, & Hinton (2015).↩︎

  181. Sutskever (2015).↩︎

  182. Goodfellow et al. (2016).↩︎

  183. Kaplan, J. et al. (2019).↩︎

  184. Rumelhart, Hinton, & Williams (1986).↩︎

  185. LeCun & Bottou (1998).↩︎

  186. Scardapane (2024).↩︎

  187. Bottou (1998).↩︎

  188. Norvig (2011).↩︎

  189. Sutton (2019).↩︎

  190. Frankle & Carbin (2018).↩︎

  191. Bengio (2009).↩︎

  192. Belkin, Hsu, Ma, & Mandal (2019).↩︎

  193. Muthukumar, Vodrahalli, Subramanian, & Sahai (2019).↩︎

  194. Nakkiran, P. et al. (2019).↩︎

  195. Chang, Li, Oymak, & Thrampoulidis (2020).↩︎

  196. Holzmüller (2020).↩︎

  197. Dar, Muthukumar, & Baraniuk (2021).↩︎

  198. Balestriero, Pesenti, & LeCun (2021).↩︎

  199. Belkin (2021).↩︎

  200. Nagarajan (2021).↩︎

  201. Bach (2022), p. 225–230.↩︎

  202. Ghosh & Belkin (2022).↩︎

  203. Singh, Lucchi, Hofmann, & Schölkopf (2022).↩︎

  204. Hastie, Montanari, Rosset, & Tibshirani (2022).↩︎

  205. Bubeck & Sellke (2023).↩︎

  206. Gamba, Englesson, Björkman, & Azizpour (2022).↩︎

  207. Schaeffer, R. et al. (2023).↩︎

  208. Yang & Suzuki (2023).↩︎

  209. Maddox, Benton, & Wilson (2023).↩︎

  210. Steinhardt (2022).↩︎

  211. Henighan, T. et al. (2023).↩︎

  212. Mishra, D. (2020). Weight Decay == L2 Regularization?↩︎

  213. S. Chen, Dobriban, & Lee (2020).↩︎

  214. Chiley, V. et al. (2019).↩︎

  215. Kiani, Balestriero, Lecun, & Lloyd (2022).↩︎

  216. Huang, L. et al. (2020).↩︎

  217. Hu, E.J. et al. (2021).↩︎

  218. Dettmers, Pagnoni, Holtzman, & Zettlemoyer (2023).↩︎

  219. Zhao, J. et al. (2024).↩︎

  220. Huh, Cheung, Wang, & Isola (2024).↩︎

  221. Fukushima & Miyake (1982).↩︎

  222. LeCun, Y. et al. (1989).↩︎

  223. LeCun, Bottou, Bengio, & Haffner (1998).↩︎

  224. Ciresan, Meier, Masci, & Schmidhuber (2012).↩︎

  225. Krizhevsky, Sutskever, & Hinton (2012).↩︎

  226. Simonyan & Zisserman (2014).↩︎

  227. He, Zhang, Ren, & Sun (2015).↩︎

  228. Haber & Ruthotto (2017).↩︎

  229. Howard, A.G. et al. (2017).↩︎

  230. R. T. Q. Chen, Rubanova, Bettencourt, & Duvenaud (2018).↩︎

  231. Tan & Le (2019).↩︎

  232. Dosovitskiy, A. et al. (2020).↩︎

  233. Tan & Le (2021).↩︎

  234. H. Liu, Dai, So, & Le (2021).↩︎

  235. Dhariwal & Nichol (2021).↩︎

  236. Liu, Y. et al. (2021).↩︎

  237. Ingrosso & Goldt (2022).↩︎

  238. Park & Kim (2022).↩︎

  239. Zhao, W.X. et al. (2023).↩︎

  240. Nakkiran, Bradley, Zhou, & Advani (2024).↩︎

  241. Firth (1957).↩︎

  242. Nirenburg (1996).↩︎

  243. Hutchins (2000).↩︎

  244. Jurafsky & Martin (2022).↩︎

  245. Z. Liu, Lin, & Sun (2023).↩︎

  246. Mikolov, Chen, Corrado, & Dean (2013), Mikolov, Yih, & Zweig (2013), and Mikolov, T. et al. (2013).↩︎

  247. Kun (2018), p. 176–8.↩︎

  248. Hochreiter & Schmidhuber (1997).↩︎

  249. Graves (2013).↩︎

  250. Sutskever, Vinyals, & Le (2014), p. 4.↩︎

  251. Zhang (1998).↩︎

  252. Zhou & Hansen (2005).↩︎

  253. Collobert, Hannun, & Synnaeve (2019).↩︎

  254. Werbos (1990).↩︎

  255. Sutskever et al. (2014).↩︎

  256. Bahdanau, Cho, & Bengio (2015).↩︎

  257. Wu, Y. et al. (2016).↩︎

  258. Stahlberg (2019).↩︎

  259. Vaswani, A. et al. (2017).↩︎

  260. Devlin, Chang, Lee, & Toutanova (2018).↩︎

  261. Liu, Y. et al. (2019).↩︎

  262. Raffel, C. et al. (2019).↩︎

  263. Lan, Z. et al. (2019).↩︎

  264. Lewis, M. et al. (2019).↩︎

  265. Radford, Narasimhan, Salimans, & Sutskever (2018).↩︎

  266. Radford, A. et al. (2019).↩︎

  267. Brown, T.B. et al. (2020).↩︎

  268. Yang, Z. et al. (2019).↩︎

  269. Zaheer, M. et al. (2020).↩︎

  270. Edelman, Goel, Kakade, & Zhang (2021).↩︎

  271. Tay, Dehghani, Bahri, & Metzler (2022).↩︎

  272. Phuong & Hutter (2022).↩︎

  273. Chowdhery, A. et al. (2022).↩︎

  274. Ouyang, L. et al. (2022).↩︎

  275. Wolfram (2023).↩︎

  276. OpenAI (2023).↩︎

  277. Mohamadi, S. et al. (2023).↩︎

  278. Zhao, W.X. et al. (2023).↩︎

  279. Golovneva, Wang, Weston, & Sukhbaatar (2024).↩︎

  280. Dao, T. et al. (2022).↩︎

  281. Gu, Goel, & Ré (2021).↩︎

  282. Merrill & Sabharwal (2022).↩︎

  283. Bulatov, Kuratov, & Burtsev (2022).↩︎

  284. Bulatov, Kuratov, & Burtsev (2023).↩︎

  285. Bertsch, Alon, Neubig, & Gormley (2023).↩︎

  286. Mialon, G. et al. (2023).↩︎

  287. Peng, B. et al. (2023).↩︎

  288. Sun, Y. et al. (2023).↩︎

  289. Gu & Dao (2023).↩︎

  290. Wang, H. et al. (2023).↩︎

  291. Ma, S. et al. (2024).↩︎

  292. Ma, X. et al. (2024).↩︎

  293. Bhargava, Witkowski, Shah, & Thomson (2023).↩︎

  294. Dao & Gu (2024).↩︎

  295. Banerjee, Agarwal, & Singla (2024).↩︎

  296. Hestness, J. et al. (2017).↩︎

  297. Church & Hestness (2019).↩︎

  298. Kaplan, J. et al. (2020).↩︎

  299. Rae, J.W. et al. (2022).↩︎

  300. Hoffmann, J. et al. (2022).↩︎

  301. Caballero, Gupta, Rish, & Krueger (2022).↩︎

  302. Muennighoff, N. et al. (2023).↩︎

  303. Pandey (2024).↩︎

  304. Bach (2024).↩︎

  305. Mahowald, K. et al. (2023).↩︎

  306. Kosinski (2023).↩︎

  307. Watson & Floridi (2019).↩︎

  308. Gurnee, W. et al. (2023).↩︎

  309. Meng, Bau, Andonian, & Belinkov (2023).↩︎

  310. McDougall, C. et al. (2023).↩︎

  311. Alain & Bengio (2016).↩︎

  312. Belinkov (2022).↩︎

  313. Gurnee & Tegmark (2023).↩︎

  314. Sutton & Barto (2018).↩︎

  315. Arulkumaran, Deisenroth, Brundage, & Bharath (2017).↩︎

  316. Cesa-Bianchi & Lugosi (2006).↩︎

  317. Silver, Singh, Precup, & Sutton (2024).↩︎

  318. Bellman (1952).↩︎

  319. Mnih, V. et al. (2013) and Mnih, V. et al. (2015).↩︎

  320. Silver, D. et al. (2016).↩︎

  321. Silver, D. et al. (2017b).↩︎

  322. Silver, D. et al. (2017a).↩︎

  323. Hart & Mas‐Colell (2000).↩︎

  324. Roughgarden (2016).↩︎

  325. Zinkevich, Johanson, Bowling, & Piccione (2007).↩︎

  326. N. Brown (2020), p. 12.↩︎

  327. Zinkevich et al. (2007), p. 4.↩︎

  328. Tammelin (2014).↩︎

  329. Tammelin, Burch, Johanson, & Bowling (2015).↩︎

  330. Burch, Moravcik, & Schmid (2019).↩︎

  331. N. Brown & Sandholm (2019a).↩︎

  332. Zinkevich et al. (2007) and Lanctot, Waugh, Zinkevich, & Bowling (2009).↩︎

  333. N. Brown (2020), p. 6.↩︎

  334. N. Brown (2020), p. 12.↩︎

  335. Lanctot et al. (2009).↩︎

  336. Neller & Lanctot (2013).↩︎

  337. Burch, Lanctot, Szafron, & Gibson (2012).↩︎

  338. Johanson, M. et al. (2012).↩︎

  339. Schmid, M. et al. (2019).↩︎

  340. Li, H. et al. (2020).↩︎

  341. Habara, Fukuda, & Yamashita (2023).↩︎

  342. Lanctot (2013).↩︎

  343. Gibson (2014).↩︎

  344. Johanson (2016).↩︎

  345. Burch (2018).↩︎

  346. Horacek (2022).↩︎

  347. Lisy & Bowling (2016), p. 2.↩︎

  348. See NashConv exploitability defined in Lanctot, M. et al. (2017).↩︎

  349. Timbers (2020), p. 3.↩︎

  350. Johanson, Waugh, Bowling, & Zinkevich (2011).↩︎

  351. Ponsen, De Jong, & Lanctot (2011).↩︎

  352. Lisy & Bowling (2016).↩︎

  353. Timbers (2020).↩︎

  354. Kuhn (1950).↩︎

  355. Southey, F. et al. (2012).↩︎

  356. Billings, Davidson, Schaeffer, & Szafron (2002).↩︎

  357. Billings, D. et al. (2003).↩︎

  358. Johanson (2013).↩︎

  359. Bowling, Burch, Johanson, & Tammelin (2015).↩︎

  360. Heinrich & Silver (2016).↩︎

  361. Moravcik, M. et al. (2017).↩︎

  362. N. Brown & Sandholm (2017).↩︎

  363. N. Brown & Sandholm (2018).↩︎

  364. N. Brown & Sandholm (2019a).↩︎

  365. N. Brown, Lerer, Gross, & Sandholm (2019).↩︎

  366. N. Brown & Sandholm (2019b).↩︎

  367. N. Brown, Bakhtin, Lerer, & Gong (2020).↩︎

  368. N. Brown (2020).↩︎

  369. Schmid, M. et al. (2021).↩︎

  370. Kovarik, V. et al. (2022).↩︎

  371. Denby (1988).↩︎

  372. Denby (1993).↩︎

  373. Spears, B.K. et al. (2018).↩︎

  374. Cranmer, Seljak, & Terao (2021).↩︎

  375. Liu, Z. et al. (2024).↩︎

  376. Jiao, L. et al. (2024).↩︎

  377. Cilibrasi & Vitanyi (2005).↩︎

  378. Hutter (2007).↩︎

  379. Rathmanner & Hutter (2011).↩︎

  380. Wolpert & Macready (1995).↩︎

  381. Wolpert (1996).↩︎

  382. Wolpert & Macready (1997).↩︎

  383. Shalev-Shwarz & Ben-David (2014), p. 60–66.↩︎

  384. McDermott (2019).↩︎

  385. Wolpert (2007).↩︎

  386. Wolpert & Kinney (2020).↩︎

  387. Mitchell (1980).↩︎

  388. Roberts (2021).↩︎

  389. Goldreich & Ron (1997).↩︎

  390. Joyce & Herrmann (2017).↩︎

  391. Lauc (2020).↩︎

  392. Nakkiran (2021).↩︎

  393. Bousquet, O. et al. (2021).↩︎

  394. Andrews (2023).↩︎

  395. Raissi, Perdikaris, & Karniadakis (2017a), p. 2.↩︎

  396. Roberts (2021), p. 7.↩︎

  397. Dennett (1991), p. TODO.↩︎

  398. Minsky & Papert (1969), p. TODO.↩︎

  399. Hornik, Stinchcombe, & White (1989).↩︎

  400. Lu, Z. et al. (2017).↩︎

  401. Lin & Jegelka (2018).↩︎

  402. Ismailov (2020).↩︎

  403. Bishop (2006), p. 230.↩︎

  404. Opper & Kinzel (1996).↩︎

  405. Opper (2001).↩︎

  406. Bahri, Y. et al. (2020).↩︎

  407. Halverson, Maiti, & Stoner (2020).↩︎

  408. Canatar, Bordelon, & Pehlevan (2020).↩︎

  409. Roberts, Yaida, & Hanin (2021).↩︎

  410. Cantwell (2022).↩︎

  411. Dinan, Yaida, & Zhang (2023).↩︎

  412. Sohl-Dickstein (2020).↩︎

  413. Aifer, M. et al. (2023).↩︎

  414. Geshkovski, Letrouit, Polyanskiy, & Rigollet (2023).↩︎

  415. Dieleman, Fauw, & Kavukcuoglu (2016).↩︎

  416. T. S. Cohen & Welling (2016).↩︎

  417. T. S. Cohen, Weiler, Kicanaoglu, & Welling (2019).↩︎

  418. Fuchs, Worrall, Fischer, & Welling (2020).↩︎

  419. Bogatskiy, A. et al. (2023).↩︎

  420. Marchetti, Hillar, Kragic, & Sanborn (2023).↩︎

  421. Battiloro, C. et al. (2024).↩︎

  422. Bérut, A. et al. (2012).↩︎

  423. Bérut, Petrosyan, & Ciliberto (2015).↩︎

  424. Smith (2019).↩︎

  425. Nielsen (2020).↩︎

  426. Amari (1998).↩︎

  427. Amari (2016).↩︎

  428. Balasubramanian (1996a).↩︎

  429. Balasubramanian (1996b).↩︎

  430. Calin & Udriste (2014).↩︎

  431. de Carvalho, Page, & Barney (2019).↩︎

  432. Lei, Luo, Yau, & Gu (2018).↩︎

  433. Gao & Chaudhari (2020).↩︎

  434. Bronstein, Bruna, Cohen, & Velickovic (2021).↩︎

  435. Fefferman, Mitter, & Narayanan (2016).↩︎

  436. Raissi et al. (2017a) and Raissi, Perdikaris, & Karniadakis (2017b).↩︎

  437. Karniadakis, G.E. et al. (2021).↩︎

  438. Howard, Mandt, Whiteson, & Yang (2021).↩︎

  439. Thuerey, N. et al. (2021).↩︎

  440. Cranmer, Pavez, & Louppe (2015).↩︎

  441. Cranmer, Brehmer, & Louppe (2019).↩︎

  442. Baydin, A.G. et al. (2019).↩︎

  443. Anderson (2008).↩︎

  444. Asch, M. et al. (2018).↩︎

  445. D’Agnolo & Wulzer (2019).↩︎

  446. Krenn, M. et al. (2022).↩︎

  447. Udrescu & Tegmark (2020).↩︎

  448. Cranmer, M. et al. (2020).↩︎

  449. Z. Liu, Madhavan, & Tegmark (2022).↩︎

  450. Lu, C. et al. (2024).↩︎

  451. Asch, M. et al. (2018).↩︎

  452. Korb (2001).↩︎

  453. Williamson (2009).↩︎

  454. Bensusan (2000).↩︎

  455. Perone (2018).↩︎

  456. Tenney, I. et al. (2019).↩︎

  457. Nissim, Noord, & Goot (2019).↩︎

  458. Skelac & Jandric (2020).↩︎

  459. Bender & Koller (2020).↩︎

  460. Elhage, N. et al. (2022).↩︎

  461. Patel & Pavlick (2022).↩︎

  462. Lovering & Pavlick (2022).↩︎

  463. Huh et al. (2024).↩︎

  464. Jamali, M. et al. (2024).↩︎

  465. Wittgenstein (2009), §43.↩︎

  466. Wittgenstein (2009), §340.↩︎

  467. Piantadosi (2023), p. 15.↩︎

  468. Wasserman (2003).↩︎

  469. Savage (1954).↩︎