Real-time driver drowsiness detection

The question

Fatigue is estimated to contribute to around 20% of UK road crashes but is officially recorded in about 4%, because unlike alcohol there is no roadside test for it. That gap is the case for automated detection.

The literature is full of drowsiness classifiers reporting high accuracy. Almost all of them report it on a held-out split of the dataset they were trained on. The question that actually determines whether any of this saves a life is different and much harder: does the detector still work in a moving vehicle, with vibration, changing light, a real face at a real angle, and a latency budget?

So the project was built to answer the deployment question, not the benchmark question — three models, and then a road test designed to try to break them.

The data

Three separate datasets for three separate tasks.

The geometric-feature model used facial-landmark-derived measures — eye aspect ratio, mouth aspect ratio, head yaw and pitch — computed per frame. The full-face eye-state CNN used 27,200 grayscale images at 224×224, split 19,040 train / 4,080 validation / 4,080 test. The cropped-eye classifier used a much larger set: 65,340 train / 16,335 validation / 3,223 test.

Two data problems shaped the design, and one of them was not caught until late.

Subject leakage is the obvious trap in drowsiness data, because consecutive frames of the same person are near-duplicates. A random split puts frame n in training and frame n+1 in test and reports an accuracy that measures nothing. The geometric-feature pipeline was therefore split by person, not by frame, so no subject appears on both sides.

The rolling CNN did not get that protection, and I did not discover this until the late stages of the project. It was found to be suffering from data leakage after the road test had already shown it failing. That sequence — deployed, failed, then diagnosed — is the wrong order, and it is the single most instructive thing in the project.

The test subject for the road evaluation was one person (my brother), in one vehicle, in daytime, at low speed, in roughly two-minute runs. That is not an evaluation sample; it is a smoke test.

Approach

Three models rather than one, chosen so that each covers the others' failure mode.

XGBoost on engineered geometric features, with a stacked ensemble (XGBoost, LightGBM, random forest, logistic-regression meta-learner, 5-fold CV) as a comparison. Chosen for the pre-drive check because it is interpretable: SHAP and feature-importance analysis can say why it called someone drowsy, which a CNN cannot, and a pre-drive assessment is exactly the context where a user will want a reason.

A custom CNN on full-face grayscale images — three convolutional blocks (8→16→32 channels, each with batch norm and max pooling) into three fully-connected layers with 0.4 dropout, label smoothing 0.1, Adam, early stopping. Deliberately small: it had to run on CPU in a live loop.

A fine-tuned ResNet-18 on cropped eyes. Transfer learning rather than training from scratch because the cropped-eye task has a large dataset but a narrow visual vocabulary, and ImageNet features transfer well to it.

Grad-CAM was used on both deep models to check where they were looking, not just whether they were right — which is how the ResNet's characteristic failure was identified.

The system was then integrated into a live Python dashboard with blink-rate metrics, drowsiness scoring and audio alerts, and driven on a road.

What I found

On held-out test data:

In the car, the ranking inverted.

The ResNet-18 eye-state tracking and the XGBoost pre-drive check both held up: eye open/closed states and blink rate tracked at full frame rate with no drop-outs, and the pre-drive check produced stable classifications. Audio alerts fired within 0.2 s of detection.

The 98.31% model — the best number on the page — failed completely. On the road, vibration and changing light produced near-100% false positives; it output "drowsy" essentially constantly. Diagnosis afterwards found data leakage in its training pipeline. The 98.31% is a real number on its own test set and a meaningless number as a statement about the world, and the road test is the only reason I know that.

The most useful genuine result is therefore the negative one: the gap between the best benchmark score and the best deployed component was the whole finding. The model that generalised was not the model that won on the test set.

Two further honest results from the deployment:

Microsleep detection degraded on darker skin tones. It correctly flagged eye closures under 0.5 s for lighter skin and missed equivalent events on darker skin. For a safety system, that is not a performance footnote — it is a failure mode that is worse than not shipping, because it fails silently for a subset of users.

The stacked ensemble was not worth its complexity. It matched standalone XGBoost on accuracy (77.4%, unchanged) for a ROC-AUC improvement of 0.51%, and calibrated slightly worse at high predicted probabilities. Its real value was diagnostic — the feature-importance and SHAP analysis identified mouth aspect ratio, head pitch and head yaw as the dominant signals — not predictive.

Interpretability paid off in a specific way: the ResNet's errors are strongly asymmetric (3 false positives against 302 false negatives on the "open" class), and Grad-CAM showed the model attending to the correct region but without strong activation — consistent with subtly-open eyes being read as closed rather than with the model looking in the wrong place.

What this doesn't show

The road test is n = 1. One driver, one car, daytime, low speed, roughly two minutes per functionality. It is enough to falsify a model that fails badly — which is exactly what it did, and that is why it was worth running — but it establishes nothing about reliability. No inference about real-world accuracy should be drawn from it, and I do not draw one.

The 98.31% figure should not be quoted as a capability. It is reported here because the contrast is the point. Any use of that number without the leakage finding attached would be misleading, and it is the reason I would not put "98% accuracy" on a CV line without the sentence that follows it.

The skin-tone failure was observed, not measured. It was identified qualitatively during the road test. There is no stratified evaluation, no per-group metric, no quantification of the disparity. Naming it is the minimum; a system like this needs a skin-tone-stratified test set before it goes anywhere near a user, and building that would be the first thing I did next.

No comparison against a physiological or vehicular baseline. The literature reviewed includes EEG-based methods reporting 85.6% accuracy and steering-pattern methods at 72.92%. This system was not evaluated head-to-head against either, so "camera-based is sufficient" is not supported by anything here.

Drowsiness labels are weak. The general drowsiness task is labelled from appearance, not from a physiological ground truth such as PERCLOS validated against EEG. A 77% accuracy against a soft label is not the same quantity as 77% against a hard one, and per-individual variation in the geometric model was substantial — person-specific factors, not the class boundary, drove much of the variance.

Latency was observed, not profiled. Alerts sounded within about 0.2 s and the interface froze roughly once per test for around 3 s, self-recovering. There is no proper latency distribution, and for a safety-critical alert the tail matters far more than the mean.

What I would do next: rebuild the rolling classifier with person-based splits from the start and re-run the road test as the acceptance gate rather than the post-hoc check; construct a skin-tone-stratified evaluation set and report per-group metrics as a release requirement; and add a spatio-temporal model (3D CNN or a sequence model over frames), since micro-sleep is a temporal event and every model here classifies single frames.