Mental Health Risk Detection Through Multimodal Analysis of Daily Digital Footprints

Abstract

Despite the high prevalence of mental health disorders, including depression, anxiety and suicidality, and their burden, these conditions are widely underdiagnosed worldwide. Current clinical evaluation depends on periodic self-reported symptom checklists, which provide only a snapshot of the patient’s functioning over a short period of time. The paper collates data from the 17 studies, which were published between 2013 and 2021, to explore the potential to integrate the three types of passive digital footprints — social media language/imagery, smartphone sensor logs and wearable physiological data — to enable continuous unobtrusive detection of mental health risks. The synthesis is structured around three modalities that are assessed separately in the literature: analysis of text-based social media, behavioural sensing from a smartphone, and physiological monitoring from wearables, and then how multimodal fusion architectures combine these signals. Results showed high variance in detection performance, ranging from a language-based model trained on Facebook data with an AUC score of 0.62 to the one trained on Twitter data with a score of 0.72, depending on the richness of the features; the best unimodal models were based on behavioral entropy from smartphone GPS and home-stay duration, showing AUC as high as 0.83; wearable-only models exhibited comparatively low discriminative power, ranging from 0.51 to 0.66 AUC; the highest F1-scores for cooperative multimodal fusion models were as high as 0.900, which corresponds to approximately a 30 percent relative error reduction over unimodal baselines. The synthesis also highlights common method or ethics challenges such as the use of self-reported or crowd-sourced ground truth, selection bias of active platform users, and unanswered privacy and consent issues in passive sensing. The results indicate that integration of multimodal data, as opposed to any particular digital footprint, may be best suited for scalable mental health risk screening in a clinically translatable way.