Publications

Objective: To introduce a novel, standardised approach to evaluating AI prediction models in balancing effectiveness, efficiency and utility, using a sepsis prediction model case study.

Materials and Methods: Retrospective patient data from electronic medical records of 7 public hospitals was used to retrain and evaluate a machine learning sepsis prediction model. Four conventional metrics—area under the receiver operating curve (AUROC), sensitivity, positive predictive value, and specificity—were compared with a novel graphical display integrating metrics of predictive accuracy (effectiveness), alert burden (efficiency) and lead time of alerts relative to clinical events (utility) for different alert thresholds.

Results: The dataset comprised 977,506 inpatient admissions. The novel methodology produced a plot of four vertically aligned graphs that enables decision-makers to identify an alert threshold that optimally balances effectiveness, efficiency and utility (EEU) at the level of an entire admission, and which differs from that derived using conventional metrics.

Discussion: Conventional evaluation metrics do not consider alert timing relative to clinical events and are often applied to different evaluation datasets (sample and admission level), introducing bias and confusion. In contrast, the EEU methodology (i) generates admission level evaluations at different alert thresholds; (ii) measures alert timing relative to clinical events; and (iii) provides a visual display that enables identification of the alert threshold that optimally balances EEU factors.

Conclusion: Evaluations of prediction models for adverse events in hospitalised patients should incorporate the EEU approach in assessing model suitability and selecting alert thresholds.

Objective: Current deteriorated patient outcome measures, death and unplanned ICU admission (UPICU), don’t consider patients who deteriorate and recover on the ward, nor correctly identify the time that significant deterioration occurs. This limits fair comparative evaluation of Early Warning Tool (EWT) performance and may degrade AI deterioration prediction algorithm accuracy. A Seriously Deteriorated Patient Indicator (SDPI) is required to overcome these limitations.

Materials and methods: Using a multi-hospital, retrospective dataset and supported by a clinician committee, we developed the SDPI by (i) identifying a well-embedded baseline EWT with superior identification of patients who would transfer to the ICU in the setting of severe illness, (ii) testing additions/variations in scoring elements to improve that tools accuracy, and (iii) selecting an SDPI threshold above which patients are labelled as seriously deteriorated.

Results: 957,445 ward episodes were included (UPICU prevalence 0.4%). The superior baseline tool (AUPRC 0.0752), was successfully augmented by 13.1% (AUPRC 0.085) through 11 adjustments. The final SDPI identified 12,323 seriously deteriorated patients (1.3% of the cohort), of which 8,701 (0.9% of cohort) recovered on the ward, identifying deteriorating patients 7.2 and 71 hours earlier than UPICU or death outcomes respectively.

Discussion: This physiologically derived, reproducible SDPI identifies deteriorated patients significantly earlier than UPICU or death and includes a significant cohort of seriously deteriorated patients who would have previously been mislabelled as ‘not deteriorated’.

Conclusion: We propose this novel SDPI as part of a composite outcome measure for fairer evaluation of EWT performance and better training of AI prediction models.

Background: The role of clinician-researchers in regional healthcare is challenging. Balancing patient care, academic research, and mentoring junior staff significantly burdens these dedicated professionals. Therefore, the Australian healthcare system must provide institutional support for improving clinicians’ academic performance.

Methods: This paper describes two digital solutions implemented in a regional Australian Hospital and Health Service. The Audit, Quality, and Innovation Review panel simplifies the approval process using digital workflows for quality assurance and audit projects, and the Research Data Laboratory provides secure access to de-identified patient data and supports data analysis.

Discussion: Unlike some countries, such as the US and UK, where financial incentives or established networks drive research integration, the Townsville Hospital and Health Service focuses on empowering clinicians to address local healthcare issues through research directly. This makes the Townsville Hospital and Health Service a standout example in Australian healthcare, highlighting the significance of specialised research infrastructure and data services for clinician-led audit projects and research. This digital health solutions approach is essential for closing the gap between research and practical application, ultimately leading to improved patient care. Importantly, as a service- embedded structure, this model may be more sustainable and effective than traditional models reliant on external funding or networks in regional settings

Objective: To identify bias in using a single machine learning (ML) sepsis prediction model across multiple hospitals and care locations; evaluate the impact of six different bias mitigation strategies and propose a generic modelling approach for developing best-performing models.

Methods: We developed a baseline ML model to predict sepsis using retrospective data on patients in emergency departments (EDs) and wards across nine hospitals. We set model sensitivity at 70% and determined the number of alerts required to be evaluated (number needed to evaluate (NNE), 95% CI) for each case of true sepsis and the number of hours between the first alert and timestamped outcomes meeting sepsis-3 reference criteria (HTS3). Six bias mitigation models were compared with the baseline model for impact on NNE and HTS3.

Results: Across 969 292 admissions, mean NNE for the baseline model was significantly lower for EDs (6.1 patients, 95% CI 6 to 6.2) than for wards (7.5 patients, 95% CI 7.4 to 7.5). Across all sites, median HTS3 was 20 hours (20–21) for wards vs 5 (5–5) for EDs. Bias mitigation models significantly impacted NNE but not HTS3. Compared with the baseline model, the best- performing models for NNE with reduced interhospital variance were those trained separately on data from ED patients or from ward patients across all sites. These models generated the lowest NNE results for all care locations in seven of nine hospitals.

Conclusions: Implementing a single sepsis prediction model across all sites and care locations within multihospital systems may be unacceptable given large variances in NNE across multiple sites. Bias mitigation methods can identify models demonstrating improved performance across most sites in reducing alert burden but with no impact on the length of the prediction window.

Background: Hyperkalaemia is one of the common electrolyte disorders among hospital patients, affected by many risk factors including medications and medical conditions. Prompt treatment is important given its impact on patient mortality and morbidity, which can lead to negative patient outcomes and healthcare resource utilisation. This study aims to describe the prevalence, characteristics, and treatment of patients admitted to hospitals with hyperkalaemia and compare findings between patients with kidney failure on maintenance haemodialysis therapy and patients without kidney failure. It also aims to identify associations between hyperkalaemia and hospital length of stay.

Methods: We undertook a retrospective cohort study on adult patients admitted to Townsville University Hospital between 1st January 2018 and 31st December 2022 (n = 99,047). Patients were included if they had a serum potassium result of 5.1 mmol/L and above during their admission/s. Statistical analysis was conducted using several methods. A Welch's t test and Chi-square test were employed to assess differences between groups of patients with kidney failure on maintenance haemodialysis therapy and those without kidney failure. For comparison among multiple groups with varying severities of hyperkalaemia, the Kruskal-Wallis test with Mann-Whitney U test and logistic regression were used.

Results: 8,775 hyperkalaemic patients were included in the study, with a mean age of 64.7 years. The prevalence of hyperkalaemia was 8.9% of patients. Risk factors for hyperkalaemia were highly prevalent among those who had the condition during their admissions. Patients with kidney failure on haemodialysis who had hyperkalaemia were, on average, 6 years younger, more often Indigenous, and experienced more severe hyperkalaemia compared to other patients without kidney failure. There was a notable difference in hyperkalaemia treatment between groups with varying degrees of hyperkalaemia severity. Hyperkalaemia was not found to be associated with prolonged hospital stay.

Conclusion: Hyperkalaemia is common among hospital admissions. Patients with kidney failure on haemodialysis are at higher risk of developing severe hyperkalaemia. Treatment for hyperkalaemia was variable and likely insufficient. Timely detection and treatment of hyperkalaemia is recommended.

Objective: To derive a comprehensive implementation framework for clinical AI models within hospitals informed by existing AI frameworks and integrated with reporting standards for clinical AI research.

Materials and Methods: (1) Derive a provisional implementation framework based on the taxonomy of Stead et al and integrated with current reporting standards for AI research: TRIPOD, DECIDE-AI, CONSORT-AI. (2) Undertake a scoping review of published clinical AI implementation frameworks and identify key themes and stages. (3) Perform a gap analysis and refine the framework by incorporating missing items.

Results: The provisional AI implementation framework, called SALIENT, was mapped to 5 stages common to both the taxonomy and the report- ing standards. A scoping review retrieved 20 studies and 247 themes, stages, and subelements were identified. A gap analysis identified 5 new cross-stage themes and 16 new tasks. The final framework comprised 5 stages, 7 elements, and 4 components, including the AI system, data pipeline, human-computer interface, and clinical workflow.

Discussion: This pragmatic framework resolves gaps in existing stage- and theme-based clinical AI implementation guidance by comprehen- sively addressing the what (components), when (stages), and how (tasks) of AI implementation, as well as the who (organization) and why (policy domains). By integrating research reporting standards into SALIENT, the framework is grounded in rigorous evaluation methodologies. The framework requires validation as being applicable to real-world studies of deployed AI models.

Conclusions: A novel end-to-end framework has been developed for implementing AI within hospital clinical practice that builds on previous AI implementation frameworks and research reporting standards.

Objective: To retrieve and appraise studies of deployed artificial intelligence (AI)-based sepsis prediction algorithms using systematic methods, identify implementation barriers, enablers, and key decisions and then map these to a novel end-to-end clinical AI implementation framework.

Materials and Methods: Systematically review studies of clinically applied AI-based sepsis prediction algorithms in regard to methodological quality, deployment and evaluation methods, and outcomes. Identify contextual fac- tors that influence implementation and map these factors to the SALIENT implementation framework.

Results: The review identified 30 articles of algorithms applied in adult hospital settings, with 5 studies reporting significantly decreased mortality post-implementation. Eight groups of algorithms were identified, each sharing a common algorithm. We identified 14 barriers, 26 enablers, and 22 decision points which were able to be mapped to the 5 stages of the SALIENT implementation framework.

Discussion Empirical studies of deployed sepsis prediction algorithms demonstrate their potential for improv- ing care and reducing mortality but reveal persisting gaps in existing implementation guidance. In the exam- ined publications, key decision points reflecting real-word implementation experience could be mapped to the SALIENT framework and, as these decision points appear to be AI-task agnostic, this framework may also be applicable to non-sepsis algorithms. The mapping clarified where and when barriers, enablers, and key deci- sions arise within the end-to-end AI implementation process.

Conclusions: A systematic review of real-world implementation studies of sepsis prediction algorithms was used to validate an end-to-end staged implementation framework that has the ability to account for key factors that warrant attention in ensuring successful deployment, and which extends on previous AI implementation frameworks.