# Benchmark Suite

From Church Wiki

Revision as of 23:12, 4 October 2009 by Daniel Roy (Talk | contribs)

This page will be used to link to a wide array of popular models from statistics, machine learning and artificial intelligence implemented in Church. The goals of this effort are:

- Improve our understanding of useful language constructs, library elements and modeling idioms useful for probabilistic modeling.
- Create a suite of models which we can use to test the correctness and performance of inference algorithms that target Church or other languages.
- Write a NIPS paper where we reproduce the entirety of NIPS models in the last K years.

## Models

- Models/Latent Dirichlet Allocation
- Models/Dirichlet Process
- Models/Dirichlet Process Mixture Model
- Models/Indian Buffet Process
- Models/Linear Regression
- Models/Gaussian Processes
- Models/Deep Belief Networks
- Models/Annotated Hierarchies
- Models/Mondrian Process
- Models/HDP PCFG

Bayes nets.

-Bayes net inference -Bayes net structure learning

Cluster models.

-mixture model (dirichlet-multinomial prior, bernoulli-beta observations) -infinite mixture model (DP prior, bernoulli-beta observations) -infinite Gaussian mixture model (DP prior, normal-normal-gamma observations) -stochastic block model -infinite relational model (discrete and continuous observations) -LDA / HDP-topic model

Stochastic transition models

-HMM -iHMM -PCFG parsing (which model? any?) -iPCFG -adaptor grammars

Identity resolution (is there a good standard problem? authors/papers?)

Multi-target tracking

Bayesian matrix factorization, SVD

Soccer-league model (AIMA2e ex 14.12)

discrimination rule learning

(PO)MDP planning

SLAM?

A SAT problem??

Metrics:

Independent measures:

-Wall-clock time -Number of compute-nodes

Dependent measures:

-Prediction accuracy (vs time/computation) -Time to convergence -Variance (and bias) across runs -Mixing time