Development and Use of Indicators for Evaluating Health Outcomes

By Eric Turer, JSI Senior Consultant

The development of our public health and medical care system is historically based on scientifically assessing the ability of specific medicines, procedures, and personal behaviors to delay or avert morbidity and to postpone mortality. As such, the empirical use of health outcomes data to evaluate health services is not a new concept, however the use of this approach to examine the relative efficacy of the organizations and providers that deliver these services is only recently coming into common use. Quality assurance activities have long been standard components of the internal monitoring of health care delivery organizations, yet standardized and externally comparable measures of health care outcomes have lagged behind.

Outcome Measures

There are several factors driving the growing adoption of outcomes measures to permit open evaluation and comparison of the quality, efficacy, and cost effectiveness of individual health care providers and organizations.

First, while not a requirement for collecting outcomes data, the increasing use of electronic systems to store detailed medical information is facilitating routine access to the data necessary for examining health care in this way. As electronic medical records become the norm, and evolve to become more open and interoperable across the delivery system, the effort to gather information decreases and the ability to make standardized comparisons between these sources grows.

The measures for examining health outcomes are also evolving, with several national organizations working to develop, refine, and publish detailed standards for collecting and comparing data to evaluate the process and outcomes of care for specific conditions.

Last, as the cost of care has grown faster than measured results, there is an increasing economic imperative to reward effective care. While 'managed care' tended to reward control of nominally inappropriate utilization, the current move toward 'pay-for-performance' seeks to promote utilization of proven effective practices through alignment of financial incentives. Further, the ability to share such information with consumers via the internet has engaged the public in the goal of making more effective health care choices.

Complexity of Outcome Indicators

While the use of health outcomes indicators holds great promise, it is also fraught with complexity and must be approached with great care and forethought. Issues with the collection, analysis, and comparison aspects of the process can confound results and produce misleading findings. For example, if data is based on a sample of patients, one must assure true randomness and sufficient sample size, while considering that a sampling strategy to describe individual providers may be different than one to describe organization-level results.

Electronic medical records permit 'universal' data to be extracted, but one often finds that variations in the way providers enter data in these systems produces unexpected gaps and inconsistencies that are not obvious in the resulting output. Systems often lack key data from other providers to evaluate the applicability of a given measure, such as excluding women having had hysterectomies on measures of Pap smear rates, or documenting HgA1c tests on diabetics done by outside specialists. Case mix and acuity can also confound comparisons between providers. A hospital that performs high-risk deliveries, for example, may exhibit a higher rate of delivery complications and mortality, even though the care may be superior to what is available elsewhere.

Recognizing the balance between the growing imperative to participate in evaluating and reporting outcomes and the potential complications in doing so effectively, organizations are carefully taking the initial steps in the process. JSI recently completed a study with the Health Resources and Services Administration (HRSA) to examine the feasibility of collecting a set of core clinical indicators across the range of their funded programs. [Learn more…]

Simultaneously, JSI was assisting HRSA's Bureau of Primary Health Care (BPHC) in implementing the reporting of new clinical measures as part of their core Uniform Data Set (UDS), submitted by all Federally Qualified Health Centers nationwide. [Learn more…]

These projects provided participants with the required training, documentation, and reporting framework, and then engaged the organizations in considering sources of data, methods of extraction, internal resources, complicating factors, and the components necessary to make the data useful. While the focus of these projects has been the initial assessment of the process for grantees to collect and report the information, and the resulting data is intended for organization-level quality improvement over time, rather than cross-organizational comparison or external reporting, they highlight the crucial steps necessary to promote broader involvement in clinical outcomes reporting.

For more information, please contact us.