Bayesian Parameter Estimation (BPE) is fundamentally different compared to MLE or MAP. Whereas the latter two solve for an optimal set of parameters $\hat{\mathbf{\theta }}$ for the model, BPE treats $\mathbf{\theta }$ as a random variable with a distribution $p(\mathbf{\theta })$.

Setup

We are given a dataset $\mathcal{D}$, which contains $n$ i.i.d. features ${\mathbf{x}}_j$. Given a new feature vector $\mathbf{x}$, we want to classify it to some class $\omega$. One way to do this is by the Bayes’ decision rule. That is, we choose class ${\omega }_j$ over class ${\omega }_i$ if

where ${\mathcal{D}}_j$ only contains features belonging to class ${\omega }_j$, and vice versa. We can’t directly solve this without assuming further structure to the underlying distribution.

So, let’s assume that the distribution $p(\mathbf{x}|{\mathcal{D}}_j)$ is fully described by a model parameterized only by $\mathbf{\theta }$, a random variable. This distribution tells us how likely we are to find $\mathbf{x}$ if it were in class ${\omega }_j$. From now, I omit the subscript on ${\mathcal{D}}_j$ for brevity. Then we observe that

This is much more manageable. We can compute $p(\mathbf{x}|\mathbf{\theta })$ by plugging $\mathbf{x}$ into our assumed model. $p(\mathbf{\theta }|\mathcal{D})$ can also be computed since

To summarize, we have devised a method that gives us a likelihood for $\mathbf{x}$, averaged over all possible parameters $\mathbf{\theta }$, weighted by the prior and likelihood of $\mathbf{\theta }$ given the class conditional data $\mathcal{D}$.

Gaussian Case

In the case that our model is a Gaussian, with a mean $\mathbf{\mu }$ with distribution $p(\mathbf{\mu })$ and a known covariance $\mathbf{\Sigma }$, BPE is quite easy to compute. In this case, our parameter set consists of just $\mathbf{\mu }$.

We assume the following:

1. $p(\mathbf{x}|\mathbf{\mu })\sim \mathcal{N}(\mathbf{\mu },\mathbf{\Sigma })$. That is, our model is valid for each class.

2. $p(\mathbf{\mu })\sim \mathcal{N}({\mathbf{\mu }}_0,{\mathbf{\Sigma }}_0)$. Here, ${\mathbf{\mu }}_0,{\mathbf{\Sigma }}_0$ are our “best guess” for the shape of each class conditional distribution, before seeing the data.

Keeping in mind that our goal is to compute $p(\mathbf{x}|\mathcal{D})$, we first need to find $p(\mathbf{\mu }|\mathcal{D})$. From Bayes’ theorem:

Plugging the Gaussian formulas:

where

Therefore, $p(\mathbf{\mu }|\mathcal{D})\sim \mathcal{N}({\mathbf{\mu }}_n,{\mathbf{\Sigma }}_n)$.

To complete the exercise, we need to find $p(\mathbf{x}|\mathcal{D})$. Since $\mathbf{x}|\mathbf{\mu }\sim \mathcal{N}(\mathbf{\mu },\mathbf{\Sigma })$, we can express $\mathbf{x}=\mathbf{\mu }+\mathbf{ϵ}$. It is evident that $\mathbf{ϵ}\sim \mathcal{N}(0,\mathbf{\Sigma })$. Then $\mathbf{x}\sim \mathcal{N}({\mathbf{\mu }}_n,{\mathbf{\Sigma }}_n+\mathbf{\Sigma })$.

So it turns out with this method that we don’t need to evaluate an integral at all!

In Summary

• $p(\mathbf{\mu })\sim \mathcal{N}({\mathbf{\mu }}_0,{\mathbf{\Sigma }}_0)$, where ${\mathbf{\mu }}_0,{\mathbf{\Sigma }}_0$ are “guessed”
• $p(\mathbf{x}|\mathbf{\mu })\sim \mathcal{N}(\mathbf{\mu },\mathbf{\Sigma })$, where $\mathbf{\mu },\mathbf{\Sigma }$ are the class conditional statistics computed from $\mathcal{D}$
• $p(\mathbf{\mu }|\mathcal{D})\sim \mathcal{N}({\mathbf{\mu }}_n,{\mathbf{\Sigma }}_n)$
• $p(\mathbf{x}|\mathcal{D})\sim \mathcal{N}({\mathbf{\mu }}_n,{\mathbf{\Sigma }}_n+\mathbf{\Sigma })$. This function is used for the Bayes Decision Rule