Muhammad Ibrahim Khalid
← All projects

Real-Time Visual Question Answering for Low-Vision Navigation on a Jetson Orin NX

COMP4901D, HKUST Mar – May 2026 with Hashim code · CUDA extensions

A camera on a Jetson Orin NX (16 GB) looks at the street, a small vision-language model answers one question ("is the crossing light green?", "are there stairs?", "is something in my way?"), and the answer is spoken aloud. For a pedestrian with low vision a late answer is close to useless, so we set a 500 ms target for the whole loop.

My role. I built the vision side and everything related to speed: the SigLIP encoder, token pooling, the evaluation framework, all on-device benchmarking, and the CUDA preprocessing kernels. Hashim worked on self-speculative decoding and a streaming text-to-speech model.

Where the latency goes

On the first complete run on the Jetson, I timed every stage. Capture and image encoding were cheap; the language model accounted for over 80% of the time per frame.

Figure 1. Pipeline and median time per frame (30 March): capture 14 ms, SigLIP 41 ms, language model 557 ms.

SigLIP produces 576 image tokens per frame, and the language model attends over all of them before it can answer. Average-pooling the tokens on their spatial grid shrinks that input directly, without retraining the encoder.

Token pooling

Select a token count to see the effect of pooling on a synthetic street scene. The measurements below each setting are from the Jetson.

36 tokens (6×6) → 346 ms mean latency, 50% correct

Results

Jetson Orin NX, FP16, label-only prompt, at most 4 generated tokens. I expected the usual trade-off between speed and accuracy; the sweep did not show one until 9 tokens.

Figure 2. Mean end-to-end latency by number of image tokens. 36 tokens is both the fastest and among the most accurate settings.
Image tokensMean latencyAccuracy"Unknown" answers
576747 ms25%46%
192443 ms39%39%
81365 ms50%32%
36346 ms50%36%
9373 ms43%36%

At 36 tokens the loop is 2.2× faster than at 576 and answers twice as many questions correctly. A plausible explanation is that a 0.5B model with a lightly trained projection layer reads fewer, smoother tokens more reliably; below 36 the pooling starts to erase the signal itself. The evaluation set is small (28 images), so the accuracy column indicates direction rather than a precise value.

Training the projection on a larger public dataset (8,102 street images) reached 80.75% accuracy on 1,621 held-out images. For the latency-critical path I also tested a small MLP classifier on the SigLIP features: about 23 ms per answer against about 241 ms through the language model (both measured on a laptop GPU).

Audio sample

Silero text-to-speech running on the Jetson (31 March): the first ten seconds of answers from a full evaluation run.

Engineering issues

  1. Quantized weights read as FP16. The AWQ loader dropped the quantization config, so packed 4-bit weights were interpreted as FP16 and every answer was garbage. AutoAWQ does not support the Jetson's Python 3.8 on aarch64, so I reverted to FP16.
  2. Streaming TTS. VibeVoice needed five compatibility patches against the Jetson's older transformers release and was still unusable on the device, so speech falls back through Piper, Silero and pyttsx3.
  3. Unstable outputs. The model returned symbols, chat-template fragments or the question itself. A label-only prompt with a four-token limit resolved most cases.
  4. Dependency conflicts. Silero's torch-hub code ships its own src package, which shadowed ours; the loader now isolates our modules while Silero initialises. A LoRA adapter that worked locally scored 0% on the Jetson because of a PEFT version mismatch.
  5. Class imbalance. 83% of the obstacle training labels were positive, so the live demo almost always answered "yes". I rebalanced the data and retrained.

CUDA preprocessing kernels

I moved token pooling and audio validation into custom CUDA kernels with PyBind11 bindings (jetson-vqa-cpp-extensions). Each kernel was faster than its Python equivalent, but the end-to-end loop was unchanged.

StagePythonCUDASpeed-up
Token pooling, 576 → 90.573 ms0.518 ms1.10×
Audio validation0.654 ms0.610 ms1.07×
Full VQA loop (12 samples)550.44 ms551.70 ms1.00×

Generation accounts for roughly 490 ms of the 550 ms loop, so sub-millisecond savings in preprocessing cannot appear end to end. The stage breakdown in Figure 1 already predicted this; the lesson I took from it is to profile before optimising.

Limitations and lessons