Papers

Transformers Can Do Bayesian Inference

S Müller, N Hollmann, SP Arango, J Grabocka… - arXiv preprint arXiv …, 2021 - arxiv.org
Computer Science paper cs.LG Suggest

… Previous work in simulation-based inference is focused on simulations for which a specific … Our method is in so far similar to simulation-based inference as both solely use samples from …

Cited by Link to paper

BibTeX

@article{2112.10510v7,
Author        = {Samuel Müller and Noah Hollmann and Sebastian Pineda Arango and Josif Grabocka and Frank Hutter},
Title         = {Transformers Can Do Bayesian Inference},
Eprint        = {2112.10510v7},
ArchivePrefix = {arXiv},
PrimaryClass  = {cs.LG},
Abstract      = {Currently, it is hard to reap the benefits of deep learning for Bayesian methods, which allow the explicit specification of prior knowledge and accurately capture model uncertainty. We present Prior-Data Fitted Networks (PFNs). PFNs leverage in-context learning in large-scale machine learning techniques to approximate a large set of posteriors. The only requirement for PFNs to work is the ability to sample from a prior distribution over supervised learning tasks (or functions). Our method restates the objective of posterior approximation as a supervised classification problem with a set-valued input: it repeatedly draws a task (or function) from the prior, draws a set of data points and their labels from it, masks one of the labels and learns to make probabilistic predictions for it based on the set-valued input of the rest of the data points. Presented with a set of samples from a new supervised learning task as input, PFNs make probabilistic predictions for arbitrary other data points in a single forward propagation, having learned to approximate Bayesian inference. We demonstrate that PFNs can near-perfectly mimic Gaussian processes and also enable efficient Bayesian inference for intractable problems, with over 200-fold speedups in multiple setups compared to current methods. We obtain strong results in very diverse areas such as Gaussian process regression, Bayesian neural networks, classification for small tabular data sets, and few-shot image classification, demonstrating the generality of PFNs. Code and trained PFNs are released at https://github.com/automl/TransformersCanDoBayesianInference.},
Year          = {2021},
Month         = {Dec},
Url           = {http://arxiv.org/abs/2112.10510v7},
File          = {2112.10510v7.pdf}
}

Share