Benchmark Suite

From Church Wiki
Revision as of 00:12, 5 October 2009 by Daniel Roy (Talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

This page will be used to link to a wide array of popular models from statistics, machine learning and artificial intelligence implemented in Church. The goals of this effort are:

  • Improve our understanding of useful language constructs, library elements and modeling idioms useful for probabilistic modeling.
  • Create a suite of models which we can use to test the correctness and performance of inference algorithms that target Church or other languages.
  • Write a NIPS paper where we reproduce the entirety of NIPS models in the last K years.

Models

Bayes nets.

-Bayes net inference
-Bayes net structure learning

Cluster models.

-mixture model (dirichlet-multinomial prior, bernoulli-beta observations)
-infinite mixture model (DP prior, bernoulli-beta observations)
-infinite Gaussian mixture model (DP prior, normal-normal-gamma observations)
-stochastic block model
-infinite relational model (discrete and continuous observations)
-LDA / HDP-topic model

Stochastic transition models

-HMM
-iHMM
-PCFG parsing (which model? any?)
-iPCFG
-adaptor grammars

Identity resolution (is there a good standard problem? authors/papers?)

Multi-target tracking

Bayesian matrix factorization, SVD

Soccer-league model (AIMA2e ex 14.12)

discrimination rule learning

(PO)MDP planning

SLAM?

A SAT problem??


Metrics:

Independent measures:

-Wall-clock time
-Number of compute-nodes

Dependent measures:

-Prediction accuracy (vs time/computation)
-Time to convergence
-Variance (and bias) across runs
-Mixing time
Personal tools