Evaluating systematic linguistic reasoning in large language models via linguistics olympiad problems
DOI:
https://doi.org/10.55630/mem.2026.55.134-149Keywords:
large language models, linguistic reasoning, rule induction, evaluation methodology, linguistics olympiadsAbstract
Current evaluation benchmarks for large language models often conflate genuine reasoning ability with memorization and statistical pattern matching. As a result, high benchmark scores do not necessarily indicate the capacity to induce abstract rules or to reason systematically from limited data. In this paper, we propose a methodological framework for evaluating linguistic reasoning in large language models based on problems from international and national linguistics olympiads. These problems are designed to be solvable without prior knowledge of the target language, requiring instead the induction of formal linguistic rules from small, self-contained datasets. We formalize linguistics olympiad problems as closed-world reasoning tasks and introduce a structured evaluation protocol that combines answer correctness, reasoning quality, hallucination detection, and generalization analysis.
References
A. Bean, S. Hellsten, H. Mayne, J. Magomere, E. A. Chi, R. Chi, S. A. Hale, and H. R. Kirk. 2024. LINGOLY: a benchmark of olympiad-level linguistic reasoning puzzles in low-resource and extinct languages, Proceedings of the 38th International Conference on Neural Information Processing Systems (NIPS ’24), Vol. 37. Curran Associates Inc., Red Hook, NY, USA, Article 825, pp. 26224–26237, 2024. https://dl.acm.org/doi/10.5555/3737916.3738741
A. R. Bhattacharya, I. Papadimitriou, K. Davidson, D. Alvarez-Melis, Investigating the interaction of linguistic and mathematical reasoning in language models using multilingual number puzzles, Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 28322–28332, 2025, https://aclanthology.org/2025.emnlp-main.1438.pdf
N. A. Chi, T. Malchev, R. Kong, R. A. Chi, L. Huang, E. A. Chi, R. T. McCoy, D. Radev, Proceedings of the 6th Workshop on Research in Computational Linguistic Typology and Multilingual NLP (SIGTYP 2024), pp. 113–119, 2024, https://aclanthology.org/2024.sigtyp-1.14.pdf
S. Goyal, S. Dan, IOLBENCH: Benchmarking LLMs on Linguistic Reasoning, arXiv preprint arXiv:2501.04249, 2025.
J. Khouja, L. Yang, R. Kearns, K. Korgul, V. Neacsu, S. Hellsten, A. Bean, H. Mayne, A. Mahdi, LINGOLY-TOO: Disentangling Reasoning from Knowledge with Templatised Orthographic Obfuscation, arXiv preprint arXiv:2503.02972, 2025.
J. Hwang, K. Tanmay, S.-J. Lee, A. Agrawal, P. P. Liang, H. Palangi, K. Ayush, Learn Globally, Speak Locally: Bridging the Gaps in Multilingual Reasoning, arXiv preprint arXiv:2507.05418, 2025.
D.-C. Lian, R.-S. Huang, P.-E. Chen, C. Lim, Y.-K. Lin, G.-Y. Tseng, T.-C. Yang, Z.-Y. Lin, P.-C. Chen, S.-K. Hsieh, LingBench++: A Linguistically- Informed Benchmark and Reasoning Framework for Multi-Step and Cross-Cultural Inference with LLMs, arXiv preprint arXiv:2507.16809, 2025.
H. Lin, X. Wang, R. Yan, B. Huang, H. Ye, J. Zhu, Z. Wang, J. Zou, J. Ma, Y. Liang, Generative Evaluation of Complex Reasoning in Large Language Models (KUMO), arXiv preprint arXiv:2504.02810, 2025.
Z. Lin, Z. Gou, T. Liang, R. Luo, H. Liu, Y. Yang, CRITICBENCH: Benchmarking LLMs for Critique-Correct Reasoning, Findings of the Association for Computational Linguistics: ACL 2024, pp. 1552–1587, 2024, https://aclanthology.org/2024.findings-acl.91.pdf
N. Majmudar, E. Filatova, Can LLMs Solve and Generate Linguistic Olympiad Puzzles?, Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 19163–19200, 2025, https://aclanthology.org/2025.emnlp-main.969.pdf
R. Ramji, K. Ramji, Inductive Linguistic Reasoning with Large Language Models, Findings of the Association for Computational Linguistics: ACL 2025, pp. 22783–22810, 2025, https://aclanthology.org/2025.findings-acl.1171.pdf
G. G. Sahin, Y. Kementchedjhieva, P. Rust, I. Gurevych, PuzzLing Machines: A Challenge on Learning From Small Data, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1241–1254, 2020, https://aclanthology.org/2020.acl-main.115.pdf
E. Sánchez, B. Alastruey, C. Ropers, P. Stenetorp, M. Artetxe, M. R. Costa-jussà, Linguini: A Benchmark for Language-Agnostic Linguistic Reasoning, arXiv preprint arXiv:2409.12126, 2024.
Y.-F. Shih, Z.-L. Lin, S.-K. Hsieh, Reasoning Over the Glyphs: Evaluation of LLM’s Decipherment of Rare Scripts, arXiv preprint arXiv:2501.17785, 2025.
A. Waldis, Y. Perlitz, L. Choshen, Y. Hou, I. Gurevych, Holmes: A Benchmark to Assess the Linguistic Competence of Language Models, Transactions of the Association for Computational Linguistics, vol. 12, pp. 1616–1647, 2024, https://aclanthology.org/2024.tacl-1.88.pdf
H. Zhu, Y. Liang, W. Xu, H. Xu, Evaluating Large Language Models for In-Context Learning of Linguistic Patterns in Unseen Low Resource Languages, Proceedings of the First Workshop on Language Models for Low-Resource Languages (LoResLM 2025), pp. 414–426, 2025, https://aclanthology.org/2025.loreslm-1.31.pdf
