READABILITY METRICS: THE CASE OF RUSSIAN EDUCATIONAL TEXTS

Marina Solnyshkins¹, Igor Guryanov², Elzara Gafiyatova³, Elena Varlamova⁴*
¹Doctor in Philology, Professor, Kazan Federal University, Russia, mesoln@yandex.ru
²Ph.D in Philology, Kazan Federal University, Russia, igor.goor@gmail.com
³Ph.D in Philology, Kazan Federal University, Russia, rg-777@yandex.ru
⁴*Ph.D in Philology, Kazan Federal University, Russia, el-var@mail.ru

Abstract

Background: When creating textbooks, test papers, tests and other methodological tools, the authors are interested in the pupils' fullest understanding of the material presented. Most of the information is contained in the text, and the way this text is perceived by readers, largely determines the quality of educational material. That is why creators of educational material should consider various audiences with different background knowledge.

Literature Review: The educational text is created based on abbreviation, paraphrase, expansion, which manifests itself in changing the parameters of the text: (a) the volume (the text can be shortened or enlarged by the inclusion of explanations or elements of another text, etc.); (b) complexity (replacement of lexical grammatical composition, splitting of complex sentence into several simple ones, etc.). The educational text is a secondary, "integral and connected, actualized, didactically organized material that possesses a lexical structure and semantics corresponding to its characteristics", which is distinguished by a special set of language facilities and a pragmatic attitude, connecting the participants of communication, is the consideration of language competence and background knowledge of the addressee.

Methods and Materials: A corpus of educational texts compiled for the research includes 15 textbooks on social studies (grades 5-11) recommended by the Russian Federal Educational Standard. Subcorpus I edited by NIKITIN contains 7 textbooks ranged from 6 to11 grades. Textbooks for 11 grades have two levels of complexity, claimed by the authors: the basic and advanced levels. The second Subcorpus edited by BOGOLUBOV contains 8 textbooks for grades 5-11, books for 11 graders have two levels of complexity – basic and advanced. The total volume of corpus is 624 thousand tokens, 40 thousand sentences. Readability of the texts was computed with Flesch readability formula adapted for the Russian language.
Results and Discussions: The highest readability in Subcorpus I belongs to the textbook for 5th grade – 61.9, which corresponds to the average level of a 9th grader and does not meet the requirements of the 5th grade. The least readable is the textbook for the 11th grade in Subcorpus II, with readability 6.6, which corresponds to the level of university graduate. We also discovered a minor difference between the basic and advanced textbooks for grades 10-11 in Subcorpus I: 13.1 and 13.09, respectively. We may suggest that complexity of these two texts is almost identical.

Conclusions: Flesh-Kincaid readability formula adapted for Russian texts proved to be an invalid tool to provide information about text complexity. We cannot rely on the formula as it cannot be applied to estimate either a semantic or syntactic difficulty of a text. The study demonstrated limitations of the existing formulas and suggests further studies of complexity of Russian educational text. We believe that a conclusive research on defining text metrics correlating with text complexity implies a thorough research of cohesion of Russian texts, their lexical and syntactic parameters as well as such quantitative metrics as the length of an average word and a sentence. Readability formulas designed on the range of the metrics enumerated above may contribute to the process of better selecting educational texts.

Keywords: educational text, text complexity, readability formulas, Flesch index, educational discourse

Acknowledgements: The research presented in Part 1 was supported by the subsidy of the Russian Government to support the Program of Competitive Growth of Kazan Federal University.

The research presented in Part 2 was financially supported by the Russian Science Foundation, grant № 18-18-00436.