Institutional Repository UIN Sunan Kalijaga YogyakartaOPTIMALISASI ARSITEKTUR RETRIEVAL-AUGMENTED GENERATION MELALUI FINE-TUNING MODEL EMBEDDING DAN LARGE LANGUAGE MODEL UNTUK ANALISIS NAHWU-SHARAF KITAB KUNINGNIM.: 24206051015Subhan GhufronauthorArabic grammatical analysis (Nahwu-Sharaf) of classical Islamic texts (kitab kuning) is a cornerstone of the pesantren education tradition in Indonesia. This process still relies on manual analysis requiring deep expertise and extensive time. This study aims to evaluate the effectiveness of fine-tuning embedding models and Large Language Models (LLMs) within a Retrieval-Augmented Generation (RAG) architecture for automated Nahwu-Sharaf analysis of kitab kuning. Twelve configurations were tested comparatively, involving two embedding models (BGE-M3 and E5-Large) and two LLMs (Llama3-8B and Mistral-7B), each in baseline and fine-tuned conditions using QLoRA on the NahwuAI dataset. Evaluation was conducted using automatic metrics (Recall@K, BLEU, ROUGE-L, F1, Cosine Similarity) and human evaluation by three Arabic grammar experts. Results showed that fine-tuned BGE-M3 outperformed as the embedding model with a Recall@1 of 0.8109 (a 37.7% increase). Fine-tuned Mistral-7B was selected as the best LLM due to its stronger generalization capability on kitab kuning texts compared to Llama3-8B, which tended to overfit to training data. The RAG + fine-tuned Mistral configuration achieved the highest F1 (0.1176) and the highest expert score (3.80 out of 5). RAG was demonstrated to provide the greatest added value when paired with a model already strengthened through fine-tuning.006.3 Artificial Intelligence (Kecerdasan Buatan)2026-08-10UIN SUNAN KALIJAGA YOGYAKARTA;FAKULTAS SAINS DAN TEKNOLOGIThesis

For work being deposited by its own author: In self-archiving this collection of files and associated bibliographic metadata, I grant Institutional Repository UIN Sunan Kalijaga Yogyakarta the right to store them and to make them permanently available publicly for free on-line. I declare that this material is my own intellectual property and I understand that Institutional Repository UIN Sunan Kalijaga Yogyakarta does not assume any responsibility if there is any breach of copyright in distributing these files or metadata. (All authors are urged to prominently assert their copyright on the title page of their work.)

For work being deposited by someone other than its author: I hereby declare that the collection of files and associated bibliographic metadata that I am archiving at Institutional Repository UIN Sunan Kalijaga Yogyakarta) is in the public domain. If this is not the case, I accept full responsibility for any breach of copyright that distributing these files or metadata may entail.

Clicking on the deposit button indicates your agreement to these terms.