In the US, health care providers are required to report evidence-based quality measures to various governmental and independent regulatory agencies. Abstracting appropriate facts from a patient’s medical record provides the data for these measures. Finding and maintaining qualified staff for this vital function is a challenge to many healthcare providers. Emerging systems and technologies in large-scale clinical repositories and AI techniques for information extraction have the potential to make the process of collecting measures more consistent, accurate and efficient. This paper presents CM-Extractor, a computerized system that automates the process of quality measures abstraction using natural language processing and a rule-based approach. An evaluation of a deployed system used for hospital inpatient cases is discussed. The results showed that the NLP performed with high accuracy across multiple types of medical documents, and users were able to significantly improve productivity. Challenges remain in the areas of availability of electronic patient data and a model for deploying and supporting solutions on a large scale.