CultureEval: Quantifying Cultural Alignment in LLMs

Problem

LLMs are widely deployed across cultures but are predominantly trained on Western, English-language data. Off-the-shelf “alignment” benchmarks treat values as universal — they don’t measure whether a model’s outputs match the value distributions of specific demographic groups.

Approach

• Built a quantitative framework in Python (pandas, NumPy, scikit-learn) on top of survey data from ~97k respondents across 96 sociocultural indicators.
• Ran PCA to derive five latent cultural dimensions, then projected LLM responses into the same space.
• Compared model outputs against group-level survey baselines using Tucker’s Congruence Coefficient and Cohen’s d.
• Evaluated Llama-2 13B, Gemma 3 12B, and Phi-4.

Results

• All three models systematically underestimated Religious-Traditional values for non-Western demographic profiles, with Cohen’s d in the 0.89 – 1.17 range — a large effect size.
• Framework is general — drop in any model with a sampling interface and any survey instrument with a comparable schema.

Tech stack

Python · pandas · NumPy · scikit-learn · HuggingFace Transformers · Llama-2 · Gemma-3 · Phi-4