Abstract
Patients who cannot vocalize often struggle to convey symptoms during clinical encounters, increasing the risk of misunderstanding and delayed care. We present LipTalk, a mobile system that combines LipTalk (visual speech recognition) with facial-expression analysis to support patient/clinician communication. The pipeline couples a 3DCNN/BiLSTM lipreader with CTC decoding and a lightweight 2DCNN affect classifier, deployed via an Android client with server-side inference. On the publicly available datasets, our LipTalk achieves 54.8% character accuracy (CAR), 22.9% character error rate (CER), 55.6% word accuracy (WAR), and 46.2% word error rate (WER); the affect module reaches 64.1% test accuracy. A simple context-aware re-ranking yields a +2.3pp gain in WAR over lip-only decoding. Prototype tests indicate sub-second end-to-end latency for short utterances, suggesting feasibility for bedside use while motivating clinical data collection and broader evaluation.