INTRODUCTION
Translating results from published clinical trials to everyday clinical practice is not a straightforward process. Observational trials generally are conducted at the beginning of investigations to determine the effectiveness of a therapeutic intervention. However, observational trials suffer from confounding by indication, when decisions for a particular treatment are based upon clinical presentation.1,2 Randomized controlled trials (RCTs) are then developed to minimize confounding by indication, as randomization ensures that both measured and unmeasured confounders are, on average, balanced between the groups of interest.2,3 Yet RCTs are not representative of the patient population as they suffer from design restraints, restrictive enrollment criteria, patient participation, and clinical disease variability, and can also lack external validity.4-7 One additional confounder of RCTs is observational bias, when the patient and/or caregiver modifies responses to care because they are aware of the study conditions. The modification of responses as a result of being observed, or the Hawthorne effect, has its origins in the telephone equipment manufacturing industry.1,7-9
HISTORY OF THE HAWTHORNE WORKS
Telecommunication equipment, including telephones, for the American Telephone & Telegraph Company (AT&T, “Ma Bell”) was manufactured at the Western Electric Company Hawthorne Plant in Illinois from the 1920s to the mid-1960s and then transferred to the Shreveport plant in Louisiana. To improve manufacturing productivity, research was conducted in a series of observational trials to measure changes in productivity following experimental alterations in the workplace environment. Following initial experiments on workplace lighting, a special assembly workroom was established to allow production behavior to be carefully monitored following additional changes in the environmental conditions. Implementation of rest and lunch breaks, different work hour shifts, and choosing one's coworkers initially improved productivity, but once observation ceased, productivity returned to near-normal levels.10 These changes in observed productivity were later termed the Hawthorne effect.9 This effect has been studied in other disciplines such as social psychology, industrial and organizational psychology, management theory, industrial sociology, psychiatry, and now in medicine.10
HAWTHORNE EFFECT IN MEDICINE
The Hawthorne effect has been reported in perioperative studies. Nakayama et al,11 Teernstra et al,12 and Kwaan et al13 had unexpected improvements in their control groups when compared to similar control groups from earlier pilot or published studies.14 These authors credited the differences in the latter studies to provider surveillance.11-13 Improvements in interventional studies have also been reported because of the Hawthorne effect.15-18 Hence, the Hawthorne effect can occur when either the patients and/or health care workers are aware of the study conditions, which poses difficulty when generalizing the results to clinical practice. Eventually, the clinical effectiveness of any new medication needs re-examination under real-world, non-Hawthorne effect conditions.1,7-9
CONFOUNDERS
Although RCTs should evenly distribute known and unknown factors between groups,7 in clinical practice, therapies are not randomized, thereby introducing confounding by indication. In a report by Benson and Hartz, who examined treatment outcomes in 19 diverse medical and surgical treatments compared with both RCT and non-RCT methods, the outcome effect sizes were similar, with only 2 of 19 analyses dissimilar to the point where differences in treatment effects fell outside of the 95% confidence intervals.19 McKee et al reported that neither method consistently gives larger estimates of treatment effect.20
Eventually, when proposing interventions from clinical trials during everyday clinical care, patient, family, and social preferences, as well as physician preferences, will influence therapeutic decisions.20 The introduction of a novel therapeutic then must rise above this background noise of confounders to be an effective signal in demonstrating improvement in patient care.
INITIAL STUDIES WITH SUGAMMADEX
Neuromuscular blocking agents are frequently used to facilitate endotracheal intubation and improve surgical working conditions during general anesthesia.21-24 Reversal of neuromuscular blockade is frequently required to facilitate return of airway muscle function, including adequate ventilation. This reversal has been achieved through the use of acetylcholinesterase inhibitors, most commonly neostigmine, since the 1950s.25,26 However, in 2015, the FDA approved the use of the novel cyclodextrin, sugammadex, in reversing rocuronium- or vecuronium-induced neuromuscular blockade.27 Based upon the speed of sugammadex in reversing neuromuscular blockade when compared to neostigmine28,29 and with initial comparative studies reporting improved operating room discharge and postanesthesia care unit recovery times,30,31 editorials32,33 proposed that the routine use of this modified gamma-cyclodextrin could provide additional time savings and improve perioperative productivity.
A 2022 study by Moss et al examined the role of the neuromuscular blocking reversal agents on operating room times.34 Although sugammadex was associated with shorter operating room times of ∼12.5 minutes, the time savings with sugammadex were largely associated with shorter surgical times.34 We conducted a post hoc analysis of reversal agents by surgical times35 and found that as surgical times for laparoscopic cholecystectomy increased, a higher percentage of sugammadex was used for reversal of rocuronium-induced neuromuscular blockade (Figure 1). Lower median and interquartile range [IQR] surgical times for neostigmine (53 [IQR 39-75] minutes) were observed when compared to sugammadex (64 [IQR 42-99] minutes) surgical times (chi-square=120, P<0.0001).36 These post hoc results suggest that the duration of the surgical procedure played a role in the selection of the neuromuscular reversal agent. If sugammadex use was delegated to longer, probably unexpected, laparoscopic cholecystectomy surgical times, then the quicker reversal effect reported for sugammadex, when compared to neostigmine,28,37-40 should translate into less variance or spread in operating room discharge times, including the number of outliers. We parceled surgical times into 5 groups, with the studentized residuals41 of the operating room discharge times plotted against the type of neuromuscular blocking reversal agent (Figure 2). Log-linear variance analysis of the residuals, a manufacturing production regression technique,42,43 is shown in the Table.
Association of surgical times in minutes (min) and neuromuscular blocking reversal agent—sugammadex in red and neostigmine in blue—chi-square=120, P<0.0001. P values <0.005 are statistically significant.36 (For readers of the print publication, a color version of this figure is available online at https://doi.org/10.31486/toj.22.5031.)
Overlay plot of studentized residuals (sugammadex in red and neostigmine in blue) of operating room (OR) discharge times by increasing surgical time periods in minutes (min). Studentized residuals41 are displayed in logarithmic scale to improve clarity of the graph. The outlier limit for studentized residuals was set to 3 in this analysis. (For readers of the print publication, a color version of this figure is available online at https://doi.org/10.31486/toj.22.5031.)
Log-Linear Variance Analysis of Operating Room Discharge Time Residuals by Neuromuscular Reversal Agent in 1,611 Patients Undergoing Laparoscopic Cholecystectomy
The overlay plot in Figure 2 suggests a pattern of smaller values of studentized sugammadex residuals when compared to the neostigmine studentized residual values. More studentized neostigmine outliers were observed than studentized sugammadex outliers (values >3), with greater numbers observed in the higher surgical time periods. The log-linear analysis of the residuals shows that the variance of sugammadex was statistically different than that of the active control neostigmine (Table). These analyses support the hypothesis34 that in cases with longer surgical times, sugammadex may allow for faster completion of surgery as the number of studentized sugammadex outliers was less (Figure 2). Nevertheless, in the study by Lee, Ahsan, Chae, et al,35 the authors explored the perioperative efficiency of this gamma-cyclodextrin for improving operating room discharge and postanesthesia care unit recovery times without favorable results.
In summary, confounding occurs during general clinical practice, and clinical investigation needs to address Hawthorne effect conditions.9 When translating results from published clinical trials, novel therapeutics need to rise above this background noise to demonstrate improvements in patient care.
ACKNOWLEDGMENTS
The authors have no financial or proprietary interest in the subject matter of this article.
- ©2022 by the author(s); Creative Commons Attribution License (CC BY)
©2022 by the author(s); licensee Ochsner Journal, Ochsner Clinic Foundation, New Orleans, LA. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (creativecommons.org/licenses/by/4.0/legalcode) that permits unrestricted use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
REFERENCES
- 1.↵VetterTR, MaschaEJ. Bias, confounding, and interaction: lions and tigers, and bears, oh my! Anesth Analg. 2017;125(3):1042-1048. doi: 10.1213/ANE.0000000000002332
- 2.↵VandenbrouckeJP. Observational research, randomized trials, and two views of medical science. PLoS Med. 2008;5(3):e67. doi: 10.1371/journal.pmed.0050067
- 3.↵ConcatoJ, ShahN, HorwitzRI. Randomized, controlled trials, observational studies, and the hierarchy of research designs. N Engl J Med. 2000;342(25):1887-1892. doi: 10.1056/NEJM200006223422507
- 4.↵RothwellPM. Factors that can affect the external validity of randomized controlled trials. PLoS Clin Trials. 2006;1(1):e9. doi: 10.1371/journal.pctr.0010009
- 5.BothwellLE, GreeneJA, PodolskySH, JonesDS. Assessing the gold standard—lessons from the history of RCTs. N Engl J Med. 2016;374(22):2175-2181. doi: 10.1056/NEJMms1604593
- 6.Chavez-MacGregorM, GiordanoSH. Randomized clinical trials and observational studies: is there a battle? [published correction appears in J Clin Oncol. 2016 Sep 20;34(27):3360]. J Clin Oncol. 2016;34(8):772-773. doi: 10.1200/JCO.2015.64.7487
- 7.↵FriedenTR. Evidence for health decision making – beyond randomized, controlled trials. N Engl J Med. 2017;377(5):465-475. doi: 10.1056/NEJMra1614394
- 8.SedgwickP, GreenwoodN. Understanding the Hawthorne effect. BMJ. 2015;351:h4672. doi: 10.1136/bmj.h4672
- 9.↵McCambridgeJ, WittonJ, ElbourneDR. Systematic review of the Hawthorne effect: new concepts are needed to study research participation effects. J Clin Epidemiol. 2014;67(3):267-277. doi: 10.1016/j.jclinepi.2013.08.015
- 10.↵WickströmG, BendixT. The "Hawthorne effect"—what did the original Hawthorne studies actually show? Scand J Work Environ Health. 2000;26(4):363-367.
- 11.↵NakayamaH, TakayamaT, OkuboT, Subcutaneous drainage to prevent wound infection in liver resection: a randomized controlled trial. J Hepatobiliary Pancreat Sci. 2014;21(7):509-517. doi: 10.1002/jhbp.93
- 12.↵TeernstraOP, EversSM, LodderJ, Stereotactic treatment of intracerebral hematoma by means of a plasminogen activator: a multicenter randomized controlled trial (SICHPA). Stroke. 2003;34(4):968-974. doi: 10.1161/01.STR.0000063367.52044.40
- 13.↵KwaanMR, WeightCJ, CardaSJ, Abdominal closure protocol in colorectal, gynecologic oncology, and urology procedures: a randomized quality improvement trial. Am J Surg. 2016;211(6):1077-1083. doi: 10.1016/j.amjsurg.2015.10.032
- 14.↵MeyhoffCS, WetterslevJ, JorgensenLN, Effect of high perioperative oxygen fraction on surgical site infection and pulmonary complications after abdominal surgery: the PROXI randomized clinical trial. JAMA. 2009;302(14):1543-1550. doi: 10.1001/jama.2009.1452
- 15.↵GongX, HeY, AnJ, Application of a computer-assisted navigation system (CANS) in the delayed treatment of zygomatic fractures: a randomized controlled trial. J Oral Maxillofac Surg. 2017;75(7):1450-1463. doi: 10.1016/j.joms.2016.10.001
- 16.BradleyJD, HeilmanDK, KatzBP, GsellP, WallickJE, BrandtKD. Tidal irrigation as treatment for knee osteoarthritis: a sham-controlled, randomized, double-blinded evaluation. Arthritis Rheum. 2002;46(1):100-108. doi: 10.1002/1529-0131(200201)46:1<100::aid-art10037>3.0.co;2-v
- 17.RolandPS, ShoupAG, SheaMC, RicheyHS, JonesDB. Verification of improved patient outcomes with a partially implantable hearing aid, the SOUNDTEC direct hearing system. Laryngoscope. 2001;111(10):1682-1686. doi: 10.1097/00005537-200110000-00002
- 18.↵ThornesB, ShannonF, GuineyAM, HessionP, MastersonE. Suture-button syndesmosis fixation: accelerated rehabilitation and improved outcomes. Clin Orthop Relat Res. 2005;(431):207-212.
- 19.↵BensonK, HartzAJ. A comparison of observational studies and randomized, controlled trials. Am J Ophthalmol. 2000;130(5):688. doi: 10.1016/S0002-9394(00)00754-6
- 20.↵McKeeM, BrittonA, BlackN, McPhersonK, SandersonC, BainC. Methods in health services research. Interpreting the evidence: choosing between randomised and non-randomised studies. BMJ. 1999;319(7205):312-315. doi: 10.1136/bmj.319.7205.312
- 21.↵LindekaerAL, Halvor SpringborgH, IstreO. Deep neuromuscular blockade leads to a larger intraabdominal volume during laparoscopy. J Vis Exp. 2013;(76):e50045. doi: 10.3791/50045
- 22.Staehr-RyeAK, RasmussenLS, RosenbergJ, Surgical space conditions during low-pressure laparoscopic cholecystectomy with deep versus moderate neuromuscular blockade: a randomized clinical study [published correction appears in Anesth Analg. 2015 Apr;120(4):957. Dosage error in article text]. Anesth Analg. 2014;119(5):1084-1092. doi: 10.1213/ANE.0000000000000316
- 23.Van WijkRM, WattsRW, LedowskiT, TrochslerM, MoranJL, ArenasGW. Deep neuromuscular block reduces intra-abdominal pressure requirements during laparoscopic cholecystectomy: a prospective observational study. Acta Anaesthesiol Scand. 2015;59(4):434-440. doi: 10.1111/aas.12491
- 24.↵BarrioJ, ErrandoCL, García-RamónJ, SellésR, San MiguelG, GallegoJ. Influence of depth of neuromuscular blockade on surgical conditions during low-pressure pneumoperitoneum laparoscopic cholecystectomy: a randomized blinded study. J Clin Anesth. 2017;42:26-30. doi: 10.1016/j.jclinane.2017.08.005
- 25.↵ZafirovaZ, DaltonA. Neuromuscular blockers and reversal agents and their impact on anesthesia practice. Best Pract Res Clin Anaesthesiol. 2018;32(2):203-211. doi: 10.1016/j.bpa.2018.06.004
- 26.↵SrivastavaA, HunterJM. Reversal of neuromuscular block [published correction appears in Br J Anaesth. 2009 Oct;103(4):622. Dosage error in article text]. Br J Anaesth. 2009;103(1):115-129. doi: 10.1093/bja/aep093
- 27.↵ThompsonCA. Sugammadex approved to reverse NMBA effects. Am J Health Syst Pharm. 2016;73(3):100. doi: 10.2146/news160009
- 28.↵JonesRK, CaldwellJE, BrullSJ, SotoRG. Reversal of profound rocuronium-induced blockade with sugammadex: a randomized comparison with neostigmine. Anesthesiology. 2008;109(5):816-824. doi: 10.1097/ALN.0b013e31818a3fee
- 29.↵GeldnerG, NiskanenM, LaurilaP, A randomised controlled trial comparing sugammadex and neostigmine at different depths of neuromuscular blockade in patients undergoing laparoscopic surgery. Anaesthesia. 2012;67(9):991-998. doi: 10.1111/j.1365-2044.2012.07197.x
- 30.↵ZhangB, HepnerDL, TranMH, FriedmanM, KornJR, MenzinJ. Neuromuscular blockade, reversal agent use, and operating room time: retrospective analysis of US inpatient surgeries. Curr Med Res Opin. 2009;25(4):943-950. doi: 10.1185/03007990902769054
- 31.↵CammuG. Sugammadex: appropriate use in the context of budgetary constraints. Curr Anesthesiol Rep. 2018;8(2):178-185. doi: 10.1007/s40140-018-0265-6
- 32.↵PatonF, PauldenM, ChambersD, Sugammadex compared with neostigmine/glycopyrrolate for routine reversal of neuromuscular block: a systematic review and economic evaluation. Br J Anaesth. 2010;105(5):558-567. doi: 10.1093/bja/aeq269
- 33.↵Fuchs-BuderT, MeistelmanC, SchreiberJU. Is sugammadex economically viable for routine use. Curr Opin Anaesthesiol. 2012;25(2):217-220. doi: 10.1097/ACO.0b013e32834f012d
- 34.↵MossAP, PowellMF, MorganCJ, TubinisMD. Sugammadex versus neostigmine for routine reversal of neuromuscular blockade and the effect on perioperative efficiency. Proc (Bayl Univ Med Cent). 2022;35(5):599-603. doi: 10.1080/08998280.2022.2079921
- 35.↵LeeC, AhsanH, ChaeH, Perioperative efficiency of sugammadex following laparoscopic cholecystectomy in clinical practice. Ochsner J. 2022; 22(4). doi: 10.31486/toj.22.0064
- 36.↵BenjaminDJ, BergerJO, JohannessonM, Redefine statistical significance. Nat Hum Behav. 2018;2(1):6-10. doi: 10.1038/s41562-017-0189-z
- 37.↵SacanO, WhitePF, TufanogullariB, KleinK. Sugammadex reversal of rocuronium-induced neuromuscular blockade: a comparison with neostigmine-glycopyrrolate and edrophonium-atropine. Anesth Analg. 2007;104(3):569-574. doi: 10.1213/01.ane.0000248224.42707.48
- 38.SparrHJ, VermeyenKM, BeaufortAM, Early reversal of profound rocuronium-induced neuromuscular blockade by sugammadex in a randomized multicenter study: efficacy, safety, and pharmacokinetics. Anesthesiology. 2007;106(5):935-943. doi: 10.1097/01.anes.0000265152.78943.74
- 39.SuyK, MoriasK, CammuG, Effective reversal of moderate rocuronium- or vecuronium-induced neuromuscular block with sugammadex, a selective relaxant binding agent. Anesthesiology. 2007;106(2):283-288. doi: 10.1097/00000542-200702000-00016
- 40.↵VanackerBF, VermeyenKM, StruysMM, Reversal of rocuronium-induced neuromuscular block with the novel drug sugammadex is equally effective under maintenance anesthesia with propofol or sevoflurane. Anesth Analg. 2007;104(3):563-568. doi: 10.1213/01.ane.0000231829.29177.8e
- 41.↵KatzMH. Delving deeper: checking the underlying assumptions of the analysis. In: Multivariable Analysis: A Practical Guide for Clinicians and Public Health Researchers. 3rd ed. Cambridge University Press; 2011:162-179.
- 42.↵HarveyAC. Estimating regression models with multiplicative heteroscedasticity. Econometrica. 1976;44(3):461-465. doi: 10.2307/1913974
- 43.↵CookRD, WeisbergS. Diagnostics for heteroscedasticity in regression. Biometrika. 1983;70(1):1-10. doi: 10.1093/biomet/70.1.1
