Hallucination Detection in Arabic Scientific Content Generated by Large Language Models
Maya Assaf
Published: 2026/08/19 · PUB-2026-000049
Abstract
AbstractThe proliferation of Large Language Models (LLMs) in scientific content generation necessitates rigorous evaluation of their factual fidelity, particularly in resource-constrained languages like Arabic. This study addresses the critical challenge of hallucination in Arabic scientific summarization by systematically investigating the influence of source-text length on the hallucination rate. We conducted a comparative experimental analysis using two distinct models: the general-purpose, instruction-tuned LLaMA-3-8B-Instruct and the Arabic-specific AraT5-Base. Scientific passages, curated from the Arabic XLSum and AraHalluEval datasets, were stratified into three length categories (Short, Medium, and Long) to model varying input complexities. The generated summaries were subjected to a dual-phase evaluation: a fine-grained manual assessment utilizing the twelve indicators of the AraHalluEval framework, and a quantitative analysis employing Semantic Entropy [Farquhar et al., 2024] to measure semantic uncertainty. This methodology provides a robust framework for quantifying the correlation between context length and hallucination frequency. The findings of this research are expected to illuminate the differential performance of general versus specialized Arabic LLMs across input lengths, offering crucial insights for developing reliable, factually consistent LLMs for high-stakes scientific applications in the Arabic language domain.
Keywords: Large Language Models (LLMs)، Hallucination Detection، Arabic NLP، Scientific Text Summarization، Semantic Entropy