Real World Validation of VEB Algorithms
Every remote monitoring system is different.
- How is the arrhythmia detected and what is the level of accuracy?
- Has the algorithm been validated by research and peer-reviewed studies?
It is important to understand that developing a deep learning algorithm in some ways is easy. However, developing a truly robust and accurate algorithm requires a great deal of expertise and a large amount of high-quality, well-curated training and validation data. Deep learning models rely on simple computational units with trainable parameters that are stacked and connected in a way that allows for modeling of complex relationships. The result is a network with millions of trainable parameters, and with so many trainable parameters deep learning algorithms are extremely sensitive to overfitting. They must be carefully designed, trained and validated in order to ensure that the algorithm performs well when presented with never-before-seen data.
How BEATLOGIC™ CAN GIVE YOU THE RELIABILITY OF A HUMAN EXPERT?
The algorithms that make up the Preventice cloud-based analysis platform, BeatLogicTM, detect cardiac arrhythmias and route ECG to skilled technicians to maximize speed of processing while minimizing false positives and false negatives. With the highest value monitoring services like telemetry, every beat of ECG is processed by multipole advanced algorithms to ensure accuracy and the highest quality reporting. BeatLogicTM achieves state-of-the-art performance by leveraging big data in combination multiple proprietary deep learning architectures. Algorithms are trained using a diverse expertly curated dataset consisting of ECGs from more than 9,000 patients to classify beats and detect 34 types of arrhythmia.
DEEP LEARNING CASE STUDY: CLASSIFYING VENTRICULAR ECTOPIC BEATS
At Preventice, our deep learning algorithms leverage our combined expertise in algorithm development and ECG interpretation to provide the highest quality reporting on factors that influence clinician decision making. For example, the detection of ectopic heart beats that originate from an abnormal focus within the ventricles of the heart (Ventricular Ectopic Beats or VEBs). In the general population, VEBs may be of little clinical consequence, but for some patients these beats signify a potentially life-threatening and treatable cardiac irregularity.
At Preventice, state-of-the-art VEB algorithm performance was achieved by leveraging big-data in combination with a novel deep learning architecture. Training was performed using more than 650,000 beats from nearly 3,500 unique patients with a diverse array of beat morphologies and noise conditions (Figure 2, Figure 3).
The deep learning network incorporated a small two-layer network, which was fed heart rates, and a deep convolutional network, which was fed 3-beat trains (Figure 3), enabling it to classify beats based on heart rate, waveform morphology and context from surrounding beats.
Within the research literature, VEB algorithm validation is typically performed using an 11-patient sub-set of the publicly available MIT-BIH database. Currently, the best performing algorithms within the literature require patient-specific training to achieve > 90% sensitivity.
While patient specific training may be useful in a research environment, it is not practical for real-world implementation where speed is required for critical notifications and thousands of patients may be monitored in a single day.
Table 1 shows how the Preventice VEB algorithm outperforms all previously published work, achieving 99.2% sensitivity while requiring no patient specific training.
RIGOROUS REAL-WORD VALIDATION DELIVERS THE PROMISE OF DEEP LEARNING
Many of the previously published deep learning beat classification algorithms have relied on the publicly available MIT-BIH database, which consists of 30-minute ECG records from 47 patients for both training and testing. The small number of patients within this database prevents researchers from demonstrating that their algorithms perform well for new patients, particularly in cases where patients are not separated for training and validation. It also prevents researchers from identifying algorithm “blind spots”, situations where the algorithm performs poorly during particular arrhythmias that were never seen during training. Blind-spots are likely to occur during critical arrhythmias as they may not be represented in small datasets.
Validation of the Preventice VEB algorithm was performed on ECG captured using the BodyGuardian® Heart device. The algorithm achieved state-of-the-art performance with 95% sensitivity and 98% specificity on the real-world validation dataset. Real-world validation using a large diverse dataset is a necessity for evaluating deep learning algorithms and standard practice for all algorithms deployed under the BeatLogic™ platform.
- Michael Mcroberts and Benjamin A. Teplitzky, Fully-automated ventricular ectopic beat classification for use with mobile cardiac telemetry, 2018 IEEE 15th International Conference on Wearable and Implantable Body Sensor Networks, BSN 2018, vol. 2018, pp. 58–61, 2018
- Benjamin A. Teplitzky, Michael McRoberts and Suneet Mittal, Real-World Validation of a Deep Learning Algorithm for Fully-Automated Premature Ventricular Beat Classification During Ambulatory External ECG Monitoring, Heart Rhythm Society's 39th Annual Scientific Sessions in Boston, May 9-12, 2018, Abstract-PO03-044