The handling of 3D orientations is a common element in many problems that arise in the estimation and control of dynamic systems. Over-parametrizations such as unit quaternions are commonly used to avoid singularities but come with the property of an invariant which needs to be preserved. By using numerical optimization methods, these invariants are subject to numeric errors and require stabilization. In this work, we adopt methods known from optimal control for the problem of state estimation. We present an optimization-based attitude estimator using the measurements of an inertial measurement unit and evaluate the performance of a first-order stabilization of the invariant by modifying the dynamics. The uncertainties of the estimator are analyzed for different configurations of the proposed stabilization. Finally, we show how the stabilization affects the estimation of parameters and justify the use of an additional equality constraint for the invariant to yield more robust and consistent results.