Anxiety is the most prominent source of stress, harmful behaviours, and psychological disorders. AI systems, usually built for maximizing performance, increase the worldwide exposition to anxiety. This foundational paper introduces Anxiety-Aware Markov Decision Processes (AA-MDPs), the first formalism rooted in fundamental psychology research for modelling the anxiety tied to policies. In addition, this paper formalizes models and practical polynomial algorithms for generating anxiety-sensitive policies. Empirical validation demonstrates that AA-MDPs policies replicate the influence of anxiety on human decision-making observed by fundamental psychology research. Last, this paper demonstrates that AA-MDPs are directly applicable for social good, through a real-world use case (Anxiety-Sensitive Itinerary Planning), the immediate applicability for augmenting any formerly-defined MDP model with anxiety-awareness, and direct tracks developing future high-impact models.