Abstract
Despite progress in sign recognition, real-time systems that can robustly handle partial or uncertain visual cues remain limited. This challenge is especially pronounced when only a small set of keywords can be reliably detected, yet the system must still generate fluent, contextually appropriate natural-language output. In this paper, we present SignGPT, a real-time American Sign Language (ASL) keyword spotting and LLM-guided natural language generation system. We extract hand and upper-body landmarks and image crops using MediaPipe. We then employ a compact visual encoder with a BiLSTM/GRU temporal head trained with Connectionist Temporal Classification (CTC) to recognize fifteen ASL keywords without frame-level alignments. The detected keyword sequence (typically 2-5 keywords) conditions a large language model (LLM) to perform contextual sentence completion, returning the top three English sentence candidates consistent with the recognized keywords and a task prompt (e.g., domain constraints, style, or register). We outline an evaluation protocol that combines keyword-level sequence accuracy with sentence-level quality assessed via human preference studies. On a held-out test set, SignGPT achieves 97.0% sentence-level accuracy with LLM guidance versus 73.5% without, demonstrating that LLM-guided contextual completion effectively bridges real-time keyword spotting and coherent ASL-to-English sentence generation.