We propose and analyze two strategies for preconditioning linear operator equations that arise in PDE constrained optimal control in the framework of conjugate gradient methods. Our particular focus is on control or state constrained problems, where we consider the question of robustness with respect to critical parameters. We construct a preconditioner that yields favorable robustness properties in this respect.