EEG Window Leakage Inflates ADHD and Parkinson's Scores

Key Takeaways

• In pediatric ADHD visual-counting EEG and adult Parkinson's disease eyes-open resting EEG, a contaminated random-window Random Forest test produced higher apparent balanced accuracy than tests that held out participants.
• Different neural models led the participant-held-out ADHD and Parkinson's cohorts, and Bonferroni-adjusted comparisons did not establish ATCNN-SA superiority.
• Participant-bootstrap estimates remained uncertain despite thousands of EEG windows in both cohorts.
• Window-length and ATCNN-SA ablation patterns differed by cohort, with no significant adjusted component-removal comparison.

An electroencephalography (EEG) classifier may encounter another overlapping segment from a familiar person rather than a recording from someone it has never seen. For people with attention-deficit/hyperactivity disorder (ADHD) or Parkinson disease, those are different prediction tasks. EEG recordings can yield many segments without yielding more independent patients. Recordings from separate pediatric and adult cohorts allowed investigators to examine the distinction.

An EEG window-leakage analysis, published in Frontiers in Neuroinformatics, was a methodological secondary analysis of public recordings. The pediatric cohort included 121 children assessed during visual counting for an ADHD-versus-control comparison. The adult cohort included 149 people with Parkinson's disease or serving as controls, recorded during eyes-open rest. Diagnosis, age and recording state differed. The study compared a deliberately contaminated stratified 80/20 random-window Random Forest audit with participant-disjoint testing using the same preprocessing, feature extractor and classifier settings.

Overlapping windows from a recording share correlated activity, and person- or session-specific patterns can persist across segments. If one person’s windows land in both training and test sets, a classifier can draw on familiar patterns when predicting a supposedly held-out window. That window-level score differs from classification of an unseen person. For both cohorts, signals were harmonized to 18 scalp channels sampled at 128 hertz, standardized within each participant and divided into half-overlapping windows. Window duration varied to assess sensitivity to segmentation.

All windows from an individual stayed in the same outer test fold during stratified eight-fold cross-validation. A stratified 15% of outer-training participants supplied internal validation, with none drawn from the test fold. Participant-disjoint neural comparisons included a one-dimensional convolutional neural network (1D-CNN), a convolutional neural network–long short-term memory hybrid (CNN-LSTM), EEGNet and ATCNN-SA. Mean window probabilities generated each person’s principal classification.

In the ADHD cohort, deliberately contaminated random-window Random Forest balanced accuracy was 96%, compared with 61% when participants were held out. The matching preprocessing and classifier settings made the split unit the manipulated factor in this audit.

The Parkinson's disease cohort showed the same split-related pattern: contaminated random-window balanced accuracy was 92%, versus 56% with participants held out.

Among participant-disjoint neural models using mean-probability aggregation, CNN-LSTM led the ADHD cohort with 73% mean subject-level balanced accuracy; EEGNet led the Parkinson's cohort at 64%. No paired comparison established ATCNN-SA superiority after within-cohort Bonferroni correction. Window-duration rankings changed across cohorts and measures, as did the effects of removing attention, recurrence or multiscale components from ATCNN-SA. No adjusted ablation comparison was significant, leaving no consistent window-length or component advantage across the settings.

The ADHD CNN-LSTM pooled out-of-fold balanced accuracy was 0.734 (95% confidence interval, 0.656–0.804) when uncertainty was estimated by resampling participants. This pooled estimate differs in calculation from the model’s mean across test folds. The Parkinson participant-bootstrap interval was also broad.

The distinct diagnostic targets and recording states replicate a split-related evaluation failure; they do not establish cross-disease transfer or external-site validation within either disease. Paired folds with overlapping training sets limit the power of corrected comparisons, so nonsignificance establishes neither model equivalence nor that an ablated component is dispensable. Fixed hyperparameters, ablations without independent retuning, uncalibrated probabilities and possible medication, comorbidity, acquisition or site confounding further constrain interpretation.

Clinician Questions

Did validation-trained aggregation improve participant-level EEG classification in ADHD and Parkinson disease?

In participant-disjoint classification, a validation-trained secondary probability aggregator improved accuracy for the ADHD 1D-CNN relative to mean-probability aggregation but lowered accuracy for all four Parkinson's disease architectures. It did not yield a universally better aggregation strategy.

Which EEG channels were lost when the ADHD and Parkinson recordings were harmonized?

Aligning the pediatric visual-task and adult resting-state recordings to a shared scalp montage excluded the ADHD recording’s Pz electrode and Parkinson electrodes without exact matching counterparts.