Chart-to-Text: CNN-RNN vs Vision-Language Models

Problem

Automatic captioning of statistical charts is a stress test for vision-language models: charts encode quantitative relationships that are easy for humans to read but hard for models to verbalize accurately. The question we asked: do modern VLMs decisively beat a tuned CNN-RNN baseline, or is the gap smaller than the marketing suggests?

Approach

• Preprocessed 27k+ Statista chart images and extracted visual features.
• Built an encoder–decoder pipeline in PyTorch; established a baseline with ResNet-50 as encoder and a single-LSTM decoder.
• Swapped the encoder to EfficientNet-B2 for stronger feature extraction at lower parameter count.
• Replaced the decoder with a dual-LSTM with coverage attention to reduce repetition and improve coverage of chart elements.
• Compared the resulting system against modern vision-language models on a held-out subset.

Results

• BLEU-4 improved from 0.18 → ~0.50 vs. the CNN-RNN baseline — nearly tripled caption quality on the same chart set.
• Tuned CNN-RNN remained competitive with VLMs on numerical-fact captions, which informed the writeup’s analysis of where VLMs do (and don’t) add value.

Tech stack

Python · PyTorch · EfficientNet-B2 · ResNet-50 · LSTM · coverage attention · BLEU · Statista dataset