Open Access

Evaluating the Adaptability of Large Language Models for Knowledge-aware Question and Answering

Jay Thakkar

Jay Thakkar

Symbiosis Centre of Applied AI (SCAAI), Symbiosis International (Deemed) UniversityPune, India
Search for this author on
Sciendo|Google Scholar
Thakkar, Jay
, 
Suresh Kolekar

Suresh Kolekar

Symbiosis Centre of Applied AI (SCAAI), Symbiosis International (Deemed) UniversityPune, India
Search for this author on
Sciendo|Google Scholar
Kolekar, Suresh
, 
Shilpa Gite

Shilpa Gite

  • Symbiosis Centre of Applied AI (SCAAI), Symbiosis International (Deemed) UniversityPune, India
  • Artificial Intelligence & Machine Learning Department, Symbiosis Institute of Technology, Symbiosis International (Deemed) UniversityPune, India
Search for this author on
Sciendo|Google Scholar
Gite, Shilpa
, 
Biswajeet Pradhan

Biswajeet Pradhan

Centre for Advanced Modelling and Geospatial Information Systems (CAMGIS), School of Civil and Environmental Engineering, Faculty of Engineering and Information Technology, University of Technology SydneyAustralia
Search for this author on
Sciendo|Google Scholar
Pradhan, Biswajeet
 and 
Abdullah Alamri

Abdullah Alamri

Department of Geology and Geophysics, College of Science, King Saud UniversityRiyadh, Saudi Arabia
Search for this author on
Sciendo|Google Scholar
Alamri, Abdullah
  
Aug 16, 2024
Evaluating the Adaptability of Large Language Models for Knowledge-aware Question and Answering's Cover Image
About this article
Previous Article
Next Article
Cite
Download Cover
Article Category: Article
Published Online: Aug 16, 2024
Received: Apr 11, 2024
DOI: https://doi.org/10.2478/ijssis-2024-0021
Keywords
© 2024 Jay Thakkar et al., published by Sciendo
This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Figure 1:

Workflow of a user knowledge level-adaptive language model system.
Workflow of a user knowledge level-adaptive language model system.

Figure 2:

Comparison of the Flesch–Kincaid grade level scores across various language models categorized by user familiarity level.
Comparison of the Flesch–Kincaid grade level scores across various language models categorized by user familiarity level.

Figure 3:

Evaluation of text complexity using the SMOG index for different language models based on user familiarity. SMOG, Simple Measure of Gobbledygook.
Evaluation of text complexity using the SMOG index for different language models based on user familiarity. SMOG, Simple Measure of Gobbledygook.

Figure 4:

Gunning Fog Index Scores demonstrating text readability across models and user familiarity levels.
Gunning Fog Index Scores demonstrating text readability across models and user familiarity levels.

Flesh–Kincaid grade level

Question Level Chat Bison Chat Bison 32k Test Bison Text Bison 32k Gemini Pro Gemini Pro Chat
What does this webpage contain? None 7.67 13.23 6.62 6.62 11.44 11.68
Basic 10.36 14.94 12.03 6.42 13.07 10.28
Completely familiar 11.76 16.4 12.79 8.18 15.54 13.83
What exactly is this documentation covering? None 7.4 9.53 11.85 11.73 12.41 12.2
Basic 10.34 14.68 15.95 14.77 13.01 14.5
Completely familiar 10.77 14.94 16.76 15.16 15.88 15.88
What are the most significant takeaways within this webpage? None 10.46 7.44 7.11 9.06 11.95 12.17
Basic 10.91 9.56 11.53 14.66 11.11 13.11
Completely familiar 10.94 11.83 15.56 15.16 14.37 14.25
What is the core purpose or focus of these webpages? None 10.4 9.8 10.8 9.42 13.81 10.86
Basic 11.2 11.13 11.7 11.76 15.38 11.63
Completely familiar 11.23 12.01 13.4 15.59 14.63 11.83
Who comprises the target audience for this Google Cloud Storage content? None 13.56 10.14 9.26 8.87 12.56 12.56
Basic 13.76 11.72 9.47 17.57 12.89 11.89
Completely familiar 16.84 12.88 11.58 20.13 13.44 12.44
What is the primary intention this documentation is aiming to achieve? None 10.13 7.37 12.91 14.95 12.04 14.61
Basic 10.52 9.53 13.53 15.74 14.4 14.4
Completely familiar 13.14 16.08 13.79 18.08 16.06 16.06
What tangible benefits can the information within these pages provide? None 9.83 11.13 7.83 8.22 9.55 9.55
Basic 10.04 11.32 8.13 10.84 10.25 10.25
Completely familiar 11.13 11.96 9.55 11.19 10.65 10.65
Are practical tips, guidelines, or advice offered through this documentation? None 7.74 13.61 12.03 13.48 12.14 12.14
Basic 10.27 14.07 12.18 14.08 14.32 14.32
Completely familiar 13.45 15.06 12.36 15.25 15.73 15.73
What real-world technology skills and knowledge can be attained from the diligent study of the pages? None 5.84 14.63 9.96 9.69 11.92 11.92
Basic 10.06 14.63 10.29 13.16 10.98 10.98
Completely fmiliar 14.12 15.42 11.92 17.35 14.09 13.5
Who represents the primary intended readership in terms of backgrounds and use cases? None 7.22 9.68 12.76 12.76 12.93 12.95
Basic 15.27 11.44 14.64 13.96 13.01 13.54
Completely familiar 16.71 14.74 15.57 16.19 14.93 14.99

SMOG

Question Level Chat Bison Chat Bison 32k Test Bison Text Bison 32k Gemini Pro Gemini Pro Chat
What does this webpage contain? None 23.5 22.5 13.02 14.55 23.9 22.5
Basic 20.12 24.8 20.74 16.4 24.11 22.86
Completely familiar 26.82 26.5 24.18 17.12 25.15 23.29
What exactly is this documentation covering? None 24.56 23.33 18.31 22.5 26.51 25.1
Basic 25.98 24.98 21.86 23.73 26.91 26.96
Completely familiar 26.19 25.25 24.6 27.03 27.25 27.25
What are the most significant takeaways within this webpage? None 20.89 17.12 19.54 21.06 25.25 25.25
Basic 24.31 22.29 25.07 27.37 26.92 26.92
Completely familiar 25.64 23.12 29.45 28.86 27.59 27.52
What is the core purpose or focus of these webpages? None 23.08 23.12 21.27 18.67 24.69 22.77
Basic 24.17 25.64 21.45 20.79 27.65 24.39
Completely familiar 25.68 26.76 23.33 26.33 28.33 25.1
Who comprises the target audience for this Google Cloud Storage content? None 21.49 20.27 21.19 21.55 20.19 20.19
Basic 22.64 21.27 22.19 22.98 20.27 20.27
Completely familiar 25.98 22.08 24.31 24.35 21.61 21.61
What is the primary intention this documentation is aiming to achieve? None 20.27 16.53 25.8 25.44 25.07 24.29
Basic 23.73 20.58 25.8 27.63 26.33 26.33
Completely familiar 27.03 25.4 25.42 28.4 28.84 28.84
What tangible benefits can the information within these pages provide? None 17.12 17.69 22.92 19.76 19.78 19.78
Basic 19.03 24.81 23.01 20.27 24.25 24.25
Completely familiar 28.52 25.74 24.69 20.89 25.95 25.95
Are practical tips, guidelines, or advice offered through this documentation? None 20.27 18.6 20.03 20.52 24.68 24.83
Basic 18.24 21.19 21.86 23.19 24.83 24.88
Completely familiar 23.73 22.64 22.92 23.63 24.88 26.68
What real-world technology skills and knowledge can be attained from diligent study of the pages? None 22.59 22.92 22.36 21.06 21.49 21.49
Basic 23.53 24.83 23 21.19 22.67 22.67
Completely familiar 22.59 25.74 23.5 31.12 26.33 24.88
Who represents the primary intended readership in terms of backgrounds and use cases? None 20.27 20.08 25.07 25.46 25.8 23.8
Basic 21.79 21.79 26.25 26.45 24.5 24.5
Completely familiar 24.76 27.03 28.36 28.25 26.8 25.8

Gunning Fog Index

Question Level Chat Bison Chat Bison 32k Test Bison Text Bison 32k Gemini Pro Gemini Pro Chat
What does this webpage contain? None 36.32 37.31 31.6 31.6 39.52 36.13
Basic 39.7 40.54 36.47 36.4 40.98 34.3
Completely familiar 40.89 42.98 40.07 39.2 42.17 38.73
What exactly is this documentation covering? None 40.27 39.33 36.55 37.98 40.59 40.59
Basic 41.28 40.41 36.77 38.97 40.67 41.31
Completely familiar 43.84 41.77 37.16 42.67 41.38 42.38
What are the most significant takeaways within this webpage? None 39.55 33.2 36.3 35.53 39.43 37.83
Basic 39.57 38.73 38.2 41.53 38 38
Completely familiar 36.57 39.23 44.06 42.17 41.53 41.45
What is the core purpose or focus of these webpages? None 37.26 39.59 35.74 35.85 38.75 37.06
Basic 38.95 40.75 38.4 36.4 41.54 37.45
Completely familiar 40.25 41.93 38.95 39.47 42.43 37.83
Who comprises the target audience for this Google Cloud Storage content? None 34.45 38 34.17 36.8 38.95 34.95
Basic 35.7 39.14 36.17 40.51 39.52 35.52
Completely familiar 40 40.2 41.68 41.26 41.92 38.92
What is the primary intention this documentation is aiming to achieve? None 36.3 34.23 38.46 39.71 39.29 40.52
Basic 40.05 36.67 40.43 41 41.65 41.65
Completely familiar 41.44 40.98 42.23 42.79 42.98 42.98
What tangible benefits can the information within these pages provide? None 32.09 35.7 32.49 39.52 41.38 41.38
Basic 34.34 37.68 37.01 35.34 39.21 39.21
Completely familiar 43.41 40.15 38.59 37.81 37.67 37.67
Are practical tips, guidelines, or advice offered through this documentation? None 33.73 37.87 36.22 38 39.08 39.08
Basic 36.3 38.13 37.25 38.89 39.46 39.46
Completely familiar 41.66 38.93 37.9 40.15 40.23 40.23
What real-world technology skills and knowledge can be attained from the diligent study of the pages? None 38.68 39.29 37.92 37.83 37.37 37.37
Basic 40.43 39.67 38.07 38.95 37.7 37.7
Completely familiar 40.62 40.25 38.24 45.13 40.4 39.28
Who represents the primary intended readership in terms of backgrounds and use cases? None 37.13 37.03 39.63 38.57 39.59 40.59
Basic 40.36 37.33 40.01 43.18 40.54 41.54
Completely familiar 41.64 42.67 44.88 43.56 41.11 41.91
Language:
English
Publication timeframe:
1 times per year
Journal Subjects:
Engineering, Introductions and Overviews, Engineering, other
Journal RSS Feed