Robots performing service tasks such as cooking and cleaning in human-centric environments require knowledge of certain environmental states in order to complete tasks successfully. While much effort has gone into developing various estimators for deriving distributions on values of unknown states, less attention has been placed on why the particular estimation problem arises. In this work, I argue that state estimation should no longer be treated as a black box. Estimating large sets of variables is computationally costly; just because a technique exists to estimate the values of certain variables does not justify its application. For robots whose ultimate mission is to complete tasks, only variables that are relevant to successful completion should be estimated. I propose to initially only track a minimal set of directly-relevant variables (attention), and gradually increase the sophistication of models on-demand (refinement), in a local fashion. This estimator refinement process is triggered by violations in expectations of task success (mismatch). This model selection framework is demonstrated through a proof-of-concept case study.