1. Consider a general optimization problem of the form max

2. Prove that the maximum likelihood estimator of the variance of a Gaussian variable is biased.

3. Regularization for Maximum Likelihood: Consider the following regularized loss minimization: 1 m _m i=1 log(1/ θ [xi ])+ 1 m _ log(1/θ)+log(1/(1−θ)) _

. _ Show that the preceding objective is equivalent to the usual empirical error had we added two pseudoexamples to the training set. Conclude that the regularized maximum likelihood estimator would be

. _ Derive a high probability bound on |ˆθ −θ_|. Hint: Rewrite this as |ˆθ −E[ˆθ ]+ E[ˆθ ]−θ_| and then use the triangle inequality and Hoeffding inequality. _ Use this to bound the true risk. Hint: Use the fact that now ˆθ ≥ 1 m+2 to relate