1. Regularization for Maximum Likelihood: Consider the following regularized loss minimization: 1 m _m i=1 log(1/ θ [xi ])+ 1 M _ log(1/θ)+log(1/(1−θ)) _
. _ Show that the preceding objective is equivalent to the usual empirical error had we added two pseudoexamples to the training set. Conclude that the regularized maximum likelihood estimator would be _ Derive a high probability bound on |ˆθ −θ_|. Hint: Rewrite this as |ˆθ −E[ˆθ ]+ E[ˆθ ]−θ_| and then use the triangle inequality and Hoeffding inequality. _ Use this to bound the true risk. Hint: Use the fact that now ˆθ ≥ 1 m+2 to relate |ˆθ −θ_| to the relative entropy.
