Real-Time Visual Question Answering for Low-Vision Navigation on a Jetson Orin NX
A camera on a Jetson Orin NX (16 GB) looks at the street, a small vision-language model answers one question ("is the crossing light green?", "are there stairs?", "is something in my way?"), and the answer is spoken aloud. For a pedestrian with low vision a late answer is close to useless, so we set a 500 ms target for the whole loop.
My role. I built the vision side and everything related to speed: the SigLIP encoder, token pooling, the evaluation framework, all on-device benchmarking, and the CUDA preprocessing kernels. Hashim worked on self-speculative decoding and a streaming text-to-speech model.
Where the latency goes
On the first complete run on the Jetson, I timed every stage. Capture and image encoding were cheap; the language model accounted for over 80% of the time per frame.
SigLIP produces 576 image tokens per frame, and the language model attends over all of them before it can answer. Average-pooling the tokens on their spatial grid shrinks that input directly, without retraining the encoder.
Token pooling
Select a token count to see the effect of pooling on a synthetic street scene. The measurements below each setting are from the Jetson.
36 tokens (6×6) → 346 ms mean latency, 50% correct
Results
Jetson Orin NX, FP16, label-only prompt, at most 4 generated tokens. I expected the usual trade-off between speed and accuracy; the sweep did not show one until 9 tokens.
| Image tokens | Mean latency | Accuracy | "Unknown" answers |
|---|---|---|---|
| 576 | 747 ms | 25% | 46% |
| 192 | 443 ms | 39% | 39% |
| 81 | 365 ms | 50% | 32% |
| 36 | 346 ms | 50% | 36% |
| 9 | 373 ms | 43% | 36% |
At 36 tokens the loop is 2.2× faster than at 576 and answers twice as many questions correctly. A plausible explanation is that a 0.5B model with a lightly trained projection layer reads fewer, smoother tokens more reliably; below 36 the pooling starts to erase the signal itself. The evaluation set is small (28 images), so the accuracy column indicates direction rather than a precise value.
Training the projection on a larger public dataset (8,102 street images) reached 80.75% accuracy on 1,621 held-out images. For the latency-critical path I also tested a small MLP classifier on the SigLIP features: about 23 ms per answer against about 241 ms through the language model (both measured on a laptop GPU).
Audio sample
Silero text-to-speech running on the Jetson (31 March): the first ten seconds of answers from a full evaluation run.
Engineering issues
- Quantized weights read as FP16. The AWQ loader dropped the quantization config, so packed 4-bit weights were interpreted as FP16 and every answer was garbage. AutoAWQ does not support the Jetson's Python 3.8 on aarch64, so I reverted to FP16.
- Streaming TTS. VibeVoice needed five compatibility patches against the Jetson's older transformers release and was still unusable on the device, so speech falls back through Piper, Silero and pyttsx3.
- Unstable outputs. The model returned symbols, chat-template fragments or the question itself. A label-only prompt with a four-token limit resolved most cases.
- Dependency conflicts. Silero's torch-hub code ships its own
srcpackage, which shadowed ours; the loader now isolates our modules while Silero initialises. A LoRA adapter that worked locally scored 0% on the Jetson because of a PEFT version mismatch. - Class imbalance. 83% of the obstacle training labels were positive, so the live demo almost always answered "yes". I rebalanced the data and retrained.
CUDA preprocessing kernels
I moved token pooling and audio validation into custom CUDA kernels with PyBind11 bindings (jetson-vqa-cpp-extensions). Each kernel was faster than its Python equivalent, but the end-to-end loop was unchanged.
| Stage | Python | CUDA | Speed-up |
|---|---|---|---|
| Token pooling, 576 → 9 | 0.573 ms | 0.518 ms | 1.10× |
| Audio validation | 0.654 ms | 0.610 ms | 1.07× |
| Full VQA loop (12 samples) | 550.44 ms | 551.70 ms | 1.00× |
Generation accounts for roughly 490 ms of the 550 ms loop, so sub-millisecond savings in preprocessing cannot appear end to end. The stage breakdown in Figure 1 already predicted this; the lesson I took from it is to profile before optimising.
Limitations and lessons
- Evaluation hygiene. An early fine-tuning set was built from augmented copies of the 28 evaluation images, so results from those runs are not reported here.
- Domain gap. Models scoring about 80% on public street images dropped to about 46% on our own photographs.
- Reproducibility. Some early results survive only in notes without raw logs; they are excluded from this page.