当前位置:科学网首页 > 小柯机器人 >详情
使用监督机器学习技术开发的预测模型方法质量较差,偏差风险较高
作者:小柯机器人 发布时间:2021/10/23 23:00:47

荷兰乌得勒支大学Constanza L Andaur Navarro团队研究了使用监督机器学习技术开发的预测模型研究中的偏倚风险。2021年10月20日出版的《英国医学杂志》发表了这项成果。

为了评估所有医学专业使用机器学习技术开发的预测模型研究的方法学质量,研究组在PubMed数据库中检索从2018年1月1日至2019年12月31日,筛选出报道使用监督机器学习进行个性化预测的多变量预测模型(诊断或预后)开发(无论是否经过外部验证)的文章。研究设计、数据来源或预测患者相关健康结果不受限制。

研究组确定研究的方法学质量,并使用偏倚预测风险评估工具(PROBAST)评估偏倚风险。该工具包含21个信号问题,专门用于识别四个领域中的潜在偏差。评估每个领域(参与者、预测因素、结果和分析)和每个研究(总体)的偏倚风险。

研究组共纳入152项研究:58项(38%)包括诊断预测模型,94项(62%)包括预后预测模型。PROBAST应用于152个开发模型和19个外部验证。在这171项分析中,148项(87%)被评定为高偏倚风险。分析领域最常被认为具有较高的偏倚风险。

在152个模型中,85个(56%)模型的每个候选预测因子的事件数不足,62个(41%)模型的缺失数据处理不当,59个(39%)模型的过度拟合评估不当。大多数模型使用适当的数据源开发(73%)和从外部验证基于机器学习的预测模型(74%)。然而,在开发的模型中,分别有60个(40%)和79个(52%)缺乏关于结果盲法和预测盲法的信息。

综上,大多数基于机器学习的预测模型的研究表明,方法质量较差,存在很大的偏差风险。导致偏倚风险的因素包括研究规模小、缺失数据处理不当以及未能处理过度拟合。为促进基于机器学习的预测模型在临床实践中的应用,必须努力改进此类研究的设计、实施、报告和验证。

附:英文原文

Title: Risk of bias in studies on prediction models developed using supervised machine learning techniques: systematic review

Author: Constanza L Andaur Navarro, Johanna A A Damen, Toshihiko Takada, Steven W J Nijman, Paula Dhiman, Jie Ma, Gary S Collins, Ram Bajpai, Richard D Riley, Karel G M Moons, Lotty Hooft

Issue&Volume: 2021/10/20

Abstract:

Objective To assess the methodological quality of studies on prediction models developed using machine learning techniques across all medical specialties.

Design Systematic review.

Data sources PubMed from 1 January 2018 to 31 December 2019.

Eligibility criteria Articles reporting on the development, with or without external validation, of a multivariable prediction model (diagnostic or prognostic) developed using supervised machine learning for individualised predictions. No restrictions applied for study design, data source, or predicted patient related health outcomes.

Review methods Methodological quality of the studies was determined and risk of bias evaluated using the prediction risk of bias assessment tool (PROBAST). This tool contains 21 signalling questions tailored to identify potential biases in four domains. Risk of bias was measured for each domain (participants, predictors, outcome, and analysis) and each study (overall).

Results 152 studies were included: 58 (38%) included a diagnostic prediction model and 94 (62%) a prognostic prediction model. PROBAST was applied to 152 developed models and 19 external validations. Of these 171 analyses, 148 (87%, 95% confidence interval 81% to 91%) were rated at high risk of bias. The analysis domain was most frequently rated at high risk of bias. Of the 152 models, 85 (56%, 48% to 64%) were developed with an inadequate number of events per candidate predictor, 62 handled missing data inadequately (41%, 33% to 49%), and 59 assessed overfitting improperly (39%, 31% to 47%). Most models used appropriate data sources to develop (73%, 66% to 79%) and externally validate the machine learning based prediction models (74%, 51% to 88%). Information about blinding of outcome and blinding of predictors was, however, absent in 60 (40%, 32% to 47%) and 79 (52%, 44% to 60%) of the developed models, respectively.

Conclusion Most studies on machine learning based prediction models show poor methodological quality and are at high risk of bias. Factors contributing to risk of bias include small study size, poor handling of missing data, and failure to deal with overfitting. Efforts to improve the design, conduct, reporting, and validation of such studies are necessary to boost the application of machine learning based prediction models in clinical practice.

DOI: 10.1136/bmj.n2281

Source: https://www.bmj.com/content/375/bmj.n2281

 

期刊信息

BMJ-British Medical Journal:《英国医学杂志》,创刊于1840年。隶属于BMJ出版集团,最新IF:27.604
官方网址:http://www.bmj.com/
投稿链接:https://mc.manuscriptcentral.com/bmj