Practical Considerations in the Measurement of Outcomes in Healthcare

Outcome Construct

Outcomes can be classified into several categories: morbidity, mortality, pain, functional status, satisfaction, and costs (1) (Table 1). Both generic and disease-specific tools can be used to measure immediate or short-term outcomes (e.g. in-hospital mortality for myocardial infarction patients), intermediate outcomes (e.g. 6-12 month functional ability after total knee replacement), or long-term outcomes (e.g. 10-year survival status for patients treated for breast cancer).

Process Construct

What is done in the care of patients and how it is done are assessed by the process construct. The technical components are often readily quantified (e.g. laboratory tests, medications prescribed, and emergency room, clinic, and hospital visits) (4). How healthcare is delivered (e.g. interpersonal style, communication skills) is more difficult to assess and quantify, but it is also important.

At times, outcomes such as intermediate and long-term events are not feasible to collect. Patient records may be incomplete because of changes in healthcare insurance, changes in providers, incomplete documentation, or other reasons. As a result, process measures that are closely related to the outcome of interest are collected and reported as “surrogate outcomes” measures. For example, the development of retinopathy in diabetic patients is an outcome that may take years to develop. Frequent retinal examinations (process) can lead to early diagnosis and treatment of retinopathy, thus minimizing or preventing further deterioration. Because long-term longitudinal data may not be available regarding the development of retinopathy, and because frequent retinal examinations (process measure) are related to the outcome of interest, frequency of annual retinal examinations in diabetic patients is often reported as a surrogate outcome measure. Although not ideal, surrogate outcomes measures can provide useful information provided they are closely related to the true outcome of interest. The ultimate goal, however, is to collect and track actual outcomes data.

Structure Construct

The structure of care construct primarily refers to characteristics of the healthcare system, providers, and patients. Although a broader listing has been previously published, system characteristics may include specialty mix and access (4). Provider characteristics may include demographics and specialty training. One key component of the structure construct is the patient characteristics (e.g. demographic, severity of primary disease, and severity of comorbid disease). In comparing outcomes across providers, institutions, or regions, there is often concern regarding the effect of patient “risk” associated with age and severity of primary and/or comorbid disease on the outcome of interest. The underlying theory is that if patients are sicker (structure), healthcare providers may alter what is done or not done (processes) for the patient (based on the patient's medical status), which in turn might impact the outcome under study. Patient characteristics such as age, severity of primary disease, and/or comorbidity are key components of the structure construct used to quantify “risk.” For many diseases, there is an established relationship between patient characteristics and healthcare outcomes. How to appropriately and accurately measure and analyze the risk of patient characteristics remains a challenge. Indicators of risk can be assessed using patient diagnoses, patient medication or laboratory profiles, and/or patient demographics, though patient demographics often do not capture enough of the risk to accurately explore potential differences in outcome. Medication and laboratory data can be used to identify patients with certain chronic and acute diseases; however, access to this data is typically limited and may be incomplete. Patient diagnosis provides useful information when assessing patient risk from severity of disease. A variety of tools have been developed with the goal of quantifying patient risk from different disease conditions. Each has unique scoring and data collection strategies. Careful consideration of patient risk factors in outcomes assessment is key to understanding variation in outcomes. Appropriate adjustment of data for patient risk factors allows medical decision-makers to compare “apples to apples.”

A diagram of a framework for the relationships between structure, process, and outcome using Diabetes Mellitus as an example is provided in Table 2. Although the long-term goal is assessment of the outcome variables listed (i.e. glucose control with glycosylated hemoglobin, retinopathy, and/or amputation), oftentimes this information is not readily available. As a result, process measures are frequently reported as surrogates of the outcome of interest.

Functional Status

One of the areas of increasing popularity in outcomes assessment is functional status measurement. A number of generic tools have been developed and used over the years, such as the Sickness Impact Profile (SIP) (10,11) and the COOP Charts (12,13). One of the current generic tools that has been widely studied in different disease conditions and different languages, and whose validity and reliability have been demonstrated is the Short Form 36 (SF 36) (9,14,15). The 36-question functional status assessment tool was developed for self-administration in patients 14 years of age and older and takes approximately 15 to 30 minutes to complete. It provides a quantitative estimate of nine areas: [1] physical function, [2] mental function, [3] role function-physical, [4] role function-mental, [5] social function, [6] energy/vitality, [7] pain, [8] health perception, and [9] change in health. We have utilized the SF 36 in a number of outcomes studies at Ochsner (16-19) and found that patients with different disease conditions appear to have unique patterns of functional status. Independent cross-sectional studies of patients with systemic lupus erythematosus (18,19) and patients with complaints of low back pain (16) revealed differences in functional status between these two patient populations and the general population of the United States (14) (Figure 2). These differences may potentially serve as a basis for medical intervention or for prompting further tracking and study. Variant patterns by other chronic disease types (e.g. hypertension, diabetes) have also been reported (20).

• Download figure
• Open in new tab
• Download powerpoint

Figure 1.

Nonmedical example regarding reliability and validity

Functional status, as assessed with the SF 36 tool in patients with certain diseases (such as hypertension), however, has not been shown to differ substantially from the functional status of patients with no chronic conditions (20). Generic functional status information may be more useful in tracking outcomes in some diseases than in others. Nevertheless, collection of functional status information is increasing. Recent developments have led to the Short Form 12 (SF 12), a subset of the SF 36 (21). In addition, physical and mental summary scales have evolved that may be useful in evaluating patient functioning (21,22). Incorporation of functional status assessment into routine practice may provide useful information in the care and management of patients with various conditions.

Predictive Modeling

In a predictive modeling study of patients with low back pain, functional status (as measured with the SF 36) was used as a predictor and not as an outcome variable (16). The outcome variable was physician-determined fit-for-work status. A number of variables were collected from patient questionnaires, self-administered upon check-in for his or her scheduled clinic appointment. Logistic regression analysis was performed on data from 107 patients with complaints of lower back pain. The final model contained the following variables: physical function scale score (SF 36), employment status, smoking status, and physical function (SF 36) by gender interaction. With the model, 90% of the NOT fit-for-work cases and 77% of the fit-for-work cases were correctly classified according to physicians' fit-for-work status classification; the overall correct classification with the model was 85%. The authors concluded that this model, utilizing four patient-reported variables, was a powerful predictor for the outcome of interest. In addition, the model may be useful as a screening mechanism to be used by other healthcare professionals not formally trained in occupational medicine but required to perform fit-for-work evaluations.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2.

Comparison of function scale scores: low back pain patients versus systemic lupus erythematosus versus general U.S. population (17)

Risk Adjustment

As described earlier, adjustment for differences in patient risk is critical in the assessment and comparison of outcomes data. In medical management, questions often arise with respect to variation in outcomes for patients undergoing different procedures for the same condition. In some cases, the medical status or “risk” of the patient drives the selection of the procedure. In order to have a valid comparison of the outcomes for variant procedures for the same condition, one must attempt to control for the potential differences in patient risk.

One study used three unique indices to assess comorbid illness in patients undergoing prostatectomy procedures for benign prostatic hyperplasia (BPH) (23): Charlson Index (24), the Kaplan Feinstein Index (25), and Index of Coexistent Disease (26,27). Although many physicians, including urologists, noted that patients undergoing transurethral prostatectomy (TURP) procedures were likely to be older and sicker than those patients undergoing open procedures, Roos and co-workers had reported an increased 5-year mortality rate for patients undergoing TURP (vs an open procedure), even in a subset of risk-adjusted data (28). This result was unexpected, and uncertainty surrounded the effect of the measure of comorbidity used on the outcome in the Roos study (28).

• Download figure
• Open in new tab
• Download powerpoint

Figure 3.

Frequency distribution of three comorbid index scores [Reprinted by permission of Blackwell Sciences, Inc.] (23)

• Download figure
• Open in new tab
• Download powerpoint

Figure 4.

ROC curves for three comorbidity indices (TURP patients only: n=253) [Reprinted by permission of Blackwell Sciences, Inc.] (23). * TPF = True Positive Fraction; + FPF = False Positive Fraction

The goal of the Ochsner study was to determine if any or all of the aforementioned indices could be used in the assessment of an observed excess mortality for patients undergoing a TURP compared with an open prostatectomy. The data collected for the generation of each index was obtained from the medical records of the patients meeting the study inclusion criteria. Several findings were reported:

• Each index resulted in a unique classification of the study population with regards to severity of comorbid illness (Figure 3). [It is important to note that there are index-dependent differences in coding and scoring, and the values are not equivalent across indices. Risk adjustment methodologies are not all alike, which could result in inconsistencies in the prediction of outcomes for different disorders.]

• A Receiver Operating Characteristic (ROC) Curve analysis which was performed to assess index sensitivity and specificity with respect to survival status for patients undergoing TURP revealed that there were no significant differences in the index curves (i.e. the areas under the curves were not significantly different) (Figure 4).

• The logistic regression analysis predicting survival differences between TURP versus open procedure patients revealed that each index had a similar effect in rendering the increased mortality associated with TURP insignificant. The reduction in risk differed depending on the index used.

The authors concluded that their analysis supported earlier work, which indicated that severity of illness accounts for higher 5-year mortality associated with patients undergoing TURP procedures (23). Further work is required to understand the relationship between index variation and risk reduction. Nevertheless, assessment of the risk associated with patient severity of illness is an important factor in outcomes assessment and analysis.