Clinimetric Properties of the Persian Handwriting Assessment Tool as a Screening Tool for Children with Specific Learning Disorder

Abstract
Background: The Persian Handwriting Assessment Tool (PHAT) evaluates speed and legibility in copying and dictation domains. The aim of the present study was to determine psychometric validation and diagnostic accuracy features of the PHAT for students with specific learning disorder.
Methods: In this cross-sectional study, handwriting of 120 participants (typically developing students, N= 60; students with specific learning disorder, N= 60; mean age (SD): 9.23 ± 0.53 years and 9.13±0.56 years, respectively) were assessed with PHAT. Receiver Operating Characteristics (ROC) curve analysis was carried out to derive validity parameters and Area Under the Curve (AUC) for diagnostic accuracy. Discriminative validity, internal consistency, test-retest reliability and absolute reliability were examined.
Results: The PHAT had significant ability to discriminate typically developing students from students with specific learning disorder in formation, spacing and alignment (AUC= 0.78-0.95) and moderate accuracy (AUC= 0.61-0.66) in text slant in both domains. Acceptable discriminative validity (P< 0.01), internal consistency (copying: α= 0.80-0.98; dictation: α= 0.83-0.98), and test-retest reliability (copying: ICC2,1= 0.75-0.98; dictation: ICC2,1= 0.78-0.98) were also obtained.
Conclusion: The current study suggests that the PHAT has satisfactory reliability, validity and high to acceptable diagnostic accuracy for students with specific learning disorder.
Keywords: Educational measurement, Handwriting, Psychometric properties, Specific learning disorder, Validation studies

Introduction
Despite technology and computers, handwriting is still an essential skill (1,2). Specific learning disorder is a neurodevelopmental and complex disorder with diverse manifestations and difficulties in processing, organizing, and retaining verbal and nonverbal information (3,4). Specific learning disorder characterized as an impairment in children’s reading, written expression (spelling, grammar, and punctuation accuracy, and clarity or organization of written expression), and mathematics cause poor and inappropriate performance below the chronological age (5). Children with specific learning disorder have handwriting difficulties which is not solely accounted for inadequate schooling (6). The prevalence of specific learning disorder is estimated 5% to 15% globally across the domains of reading, writing, and mathematics (5,7).
Children with specific learning disorder have major difficulties in educational success and academic functioning such as reading, writing, and spelling. All these can have negative effects on emotions, behavior, self-efficacy, and self-esteem (8). Consequently, these children become overwhelmed easily in addition to frustration, depression, and anxiety (9). According to literature, this neurodevelopmental disorder persists for many children and can cause lifelong impairments in occupational performance (10,11), therefore early referral and diagnosis are necessary for the provision of timely care (12). By the referral of children with specific learning disorder to occupational therapy, detailed assessment of occupational profile, occupational performance, and standard assessments of handwriting should be implemented (13). Children with specific learning disorder who are at risk or have poor handwriting should be screened as soon as possible (2). The best time to screen handwriting performance is grade 2 and the first half of grade 3 (12, 14). Approximately 30 to 60% of grade 2 and grade 3 children’s time is spent on fine manual activities, with handwriting being the most common (15). Literature indicates that children with specific learning disorder are qualitatively distinct from typically developing peers in fine manual dexterity skills like handwriting. Deficits in fine manual dexterity leads to participation restriction and impacts academic achievement (16,17). 
Handwriting as a key skill contributes to language and culture, and there are several tools [e.g., Evaluation Tool of Children’s Handwriting (ETCH), Handwriting Legibility Scale (HLS), Chinese Handwriting Analysis System (CHAS)] available for handwriting evaluation. The ETCH is designed to evaluate handwriting legibility and speed in English language for children with learning disabilities, mild developmental delay, and mild neuromuscular impairments. This tool includes a variety of writing tasks such as near-point copying (18,19). The HLS was designed to assess global legibility in typically developing children and children with developmental coordination disorder. This scale can be used in children with different languages and different writing styles, but due to its length, it can lead to fatigue in the child (20). The CHAS is designed to quantify the process and production of handwriting in Chinese language. Ergonomic and biomechanical features cannot be determined by this system (21).
Persian language is now the official language of over 110 million people worldwide. Persian scripts are cursive, have a right-to-left writing direction, and writing method of letters is based on their position in a word (22,23). Hence, suitable assessment tools should be used for this language. The Persian Handwriting Assessment Tool (PHAT) developed by Havaei et al is designed to assess legibility and speed variables in copying and dictation domains for grade 2 and grade 3 (8-10 years old) typically developing children. Acceptable psychometric properties have been reported for typically developing primary school-aged children (24-26). So far, this quickly-administered screening tool is the only available tool for early detection of handwriting difficulty in Persian-speaking children. Before administering this tool in specific learning disorder population, its psychometric properties should be determined so that it can be considered as a useful tool in future for research and clinical application. Valid and reliable evaluation tools provide objective measurement of handwriting output and are used to compare children, before and after treatment, as well as research. Therefore, the current study aimed at determining clinimetric properties (i.e. cut-off point, sensitivity, specificity, discriminative validity, internal consistency, test-retest reliability, and absolute reliability) of the PHAT as a screening tool in children with specific learning disorder.

Materials and Methods
Study design and participants
In this cross-sectional study, typically developing children from governmental schools and children with specific learning disorder from learning disorder centers (i.e. day centers for children with specific learning disorder which provide rehabilitation services) were approached for inclusion in Tehran, Iran from 2018 to 2019. All parents of children were aware of the study, and their willingness to take part in the study was considered. Of all the children, 120 children aged 8-10 years (60 typically developing children; 60 children with specific learning disorder) were recruited. Children were selected based on random cluster sampling method (27). Three educational districts were selected as sampling clusters. Clusters varied in terms of access to educational resources, services, and cultural context. Eight governmental schools and two learning disorder centers were randomly chosen from the mentioned districts. Children were selected randomly from schools and centers based on the inclusion criteria. The inclusion criteria for typically developing children were being Persian and monolingual, no documented physical impairment that affects handwriting and no cognitive disability based on demographic questionnaire and school records. The inclusion criteria for children with specific learning disorder were diagnosis by psychiatrists and neurologists via clinical evaluation (Wechsler Intelligence Scale for Children-Revised, ACID-profile, Peabody Individual Achievement Test-Revised and, Aston Index Battery) and interview, the ability to write at least one word correctly (i.e. spelling and orthographic error) in dictation domain, having dysgraphia alongside dyslexia or dyscalculia, and absence of Down syndrome, Attention-Deficit Hyperactivity Disorder, cerebral palsy, epilepsy, Autism and other behavioral or psychiatric disorders. Children were matched based on their gender and educational grade. According to the Wisconsin Card Sort Test (28,29), executive control of the two groups was not significantly different (t=1.24; df=118; p=0.22). Sample size was determined based on rule of thumb (30). Department of Education gave permission to conduct the study in schools and learning disorder centers. Written informed consent was obtained from each participant’s parent before their inclusion. 

Materials
Persian Handwriting Assessment Tool (PHAT)
This tool evaluates handwriting legibility (formation, spacing, alignment, size, and text slant) and speed in copying and dictation domains. The instruments required for this assessment tool consist of an HB lead pencil, an eraser, and a clipboard on an anti-slip cover. Each participant was asked to sit behind a desk that was appropriate based on the participant’s height. In copying domain, the children were asked to read the 12 words (خوشگل [pronounced “khoshgel”], نظافت [pronounced “nezafat”], لطیفه [pronounced “latifeh”], راضی [pronounced “razi”], درس [pronounced “dars”], ذهن [pronounced “zehn”], پنجره [pronounced “panjareh”], صبح [pronounced “sobh”], آغاز [pronounced “aghaz”], مبعث [pronounced “mab’ath”], قوچ [pronounced “ghooch”], کم [pronounced “kam”]) written at the top of the paper and then immediately write them on the lines at the bottom of the paper as their usual good handwriting. Time was recorded during the completion of the copy assignment. Time recorded in copy assignment was used to calculate the number of letters written per minute by the following formula: . In dictation domain, 12 words (آدم [pronounced “adam”], اخلاق [pronounced “akhlagh”], بازی [pronounced “bazi”], عسل [pronounced “asal”], صبحانه [pronounced “sobhaneh”], قارچ [pronounced “gharch”], مریض [pronounced “mariz”], پژمرده [pronounced “pazhmordeh”], هفت [pronounced “haft”], کاغذ [pronounced “kaghaz”], گنجشک [pronounced “gonjeshk”], وطن [pronounced “vatan”] were pronounced loudly and expressively by the occupational therapist. Enough time was given to each child to write the words. Orthographic error was recorded in dictation assignment. Formation (shape, indentation, and roundness), spacing (space between letters and adjacent words), alignment (being on line without angle), text slant (whole text angle relative to the line), and size in both domains were scored with a five-point Likert scale. Scoring procedures and interpretations are illustrated in table 1. This tool was administered individually and had sufficient psychometric properties. This test takes approximately 15 minutes to complete (25,26).

Wisconsin Card Sort Test (WCST)
This computer test was utilized to evaluate reasoning ability, set-shifting, and abstract reasoning. The WCST includes 64 stimulus cards differing in number, color, and shape on a computer screen. The participant should match the cards according to color, number or shape on the cards using a mouse. The rule for the correct match is unknown and the participant must discover it via “CORRECT” and “INCORRECT” feedback on the screen after each match is made. After maintaining the rule in ten trials, the rule changes and the participant must change the sorting strategy. Scores are generated by the number of errors made in attaining the new rule: (a) preservative error (i.e. number of errors made after the rule changed); (b) failure to maintain the rule; (c) correctly maintaining a rule through ten trials. This test takes 10-15 minutes to complete (31).

Procedure
The PHAT was administrated and scored by an experienced (5 years of experience in pediatric field) occupational therapist. All participants were assessed in one day (8 a.m. to 12 p.m.) and in a well-lit room with the least environmental noises. To assess test-retest reliability, seven-day time interval was considered between test and retest. According to literature, short time intervals between test and retest are more appropriate when examining children (32,33). The average time for the entire assessments was 30 to 45 minutes depending on the child’s speed and performance.

Data analysis
Statistical analysis was carried out with SPSS version 16.0 and MedCalc version 14.8.1. Receiver Operating Characteristics (ROC) curve analysis was done to obtain cut-off points based on Youden Index J and validity parameters [sensitivity, specificity, Positive Predictive Value (PPV), and Negative Predictive Value (NPV)] to illustrate the diagnostic ability of the PHAT. Area Under the Curve (AUC) values were determined as follows: values of 0-0.49, 0.50-0.70, 0.71-0.89 and 0.90-1 indicates low, acceptable, moderate, and high AUC, respectively (34,35). In order to establish discriminative validity, the scores obtained in legibility components and speed variables of the two groups were compared using an Independent T-test with less than 0.05 significance level. Effect size (ES) for each variable was calculated with Cohen’s d ES. Cohen’s d ES value of 0.2, 0.5, and 0.8 indicates small, moderate, and large magnitude of difference between groups, respectively (36). Internal consistency and test-retest reliability were calculated using Cronbach’s alpha coefficient and Intra-Class Correlation (ICC2,1), respectively. The Cronbach’s Alpha coefficient is considered acceptable at a level of 0.70 or higher (37). The ICC2, 1 (two-way random, absolute agreement) was calculated as                     (MSR= mean square for rows; MSE: mean square for error; MSC: mean square for columns; N= number of subjects; K= number of raters/measurements). Values of 0.70 or greater indicate acceptable test-retest reliability (38). Absolute reliability was calculated to determine measurement precision using Standard Error of Measurement (SEM)                                  and Minimal Detectable Change (MDC). The SEM is calculated as follows: and values lower than 10% of the maximum score of the measurement are considered acceptable. The MDC values as the minimum change beyond the measurement error were also calculated as follows: at 95% confidence interval (27).    

Table 1. Demographic characteristics of TD students and students with SLD and scoring procedures of Persian Handwriting Assessment Tool (PHAT)

Note. Mean score of all the 12 words in copy and dictation domains was considered as the participant’s score for each variable;

SD: Standard Deviation; TD: Typically developing; SLD: Specific Learning Disorder.

Table 2. Validity values of Persian Handwriting Assessment Tool (PHAT) for discriminating typically developing students and students with SLD, at cut points maximizing sensitivity

Note. SLD: Specific Learning Disorder; PHAT: Persian Handwriting Assessment Tool; CI: Confidence Interval; AUC: Area Under the Curve.

Table 3. Validity and Reliability of the Persian Handwriting Assessment Tool (PHAT)

 TD: Typically developing; SLD: Specific Learning Disorder; PHAT: Persian Handwriting Assessment Tool; ICC: Intra-Class Correlation; CI: Confidence Interval; LN: Letter number; min: minutes; ES: Effect Size; SEM: Standard Error of Measurement; MDC: Minimal Detectable Change

*Note that all P-values were significant (P < 0.01).

Results
Mean ± SD age of typically developing children and children with specific learning disorder was 9.23 ± 0.53 years and 9.13 ± 0.56 years, respectively. Detailed demographic characteristics of the two groups are depicted in table 1.

Diagnostic accuracy
The word formation, spacing, and alignment variables of the PHAT conveyed clinically significant ability to discriminate typically developing children from children with specific learning disorder in copying (formation: AUC=0.94, sensitivity=91.67%, specificity=86.67%; spacing: AUC=0.92, sensitivity=86.67%, specificity=83.33%; alignment: AUC=0.80, sensitivity=76.67%, specificity=73.33%) and dictation (formation: AUC=0.95, sensitivity=81.67%, specificity= 96.67%; spacing: AUC=0.91, sensitivity=68.33%, specificity=93.33%; alignment: AUC=0.78, sensitivity=70%, specificity=80%) domains. In both domains, text slant had acceptable diagnostic accuracy (copying: AUC=0.66, sensitivity=38.33%, specificity=95%; dictation: AUC=0.61, sensitivity=30%, specificity=93.33%) (Table 2).

Discriminative validity
 The difference in each legibility and speed variables was significant (P<0.01) in copying and dictation domains for typically developing children and children with specific learning disorder. In both domains, formation (copying: ES=1.46; dictation: ES=1.50), spacing (copying: ES=1.40; dictation: ES=1.34), and alignment (copying: ES=1.02; dictation: ES=0.97) had large ES. Text slant had moderate ES in copying (ES=0.79) and dictation (ES=0.60) domains. Size in both domains (copying: ES=0.43; dictation: ES=0.45) had small ES. Orthographic error (ES=1.21) in dictation domain and speed variables (second: ES=1.20; letter number per minute: ES=1.35) in copying domain had large ES (Table 3).

Reliability
Cronbach’s alpha ranging from 0.80 to 0.98 in copying domain and 0.83 to 0.98 in dictation domain indicates acceptable internal consistency. The ICC2,1 for each legibility component and speed variables was 0.75 to 0.98 in copying domain and 0.78 to 0.98 in legibility components and orthographic error in dictation domain. These results indicate acceptable test-retest reliability. The values of the SEM and MDC in both copying (SEM: 0.049-1.139; MDC: 0.135-3.147) and dictation (SEM: 0.041-0.114; MDC: 0.113-0.315) domains were in a sufficient range (lower than 10% of the maximum score) indicating acceptable absolute reliability (Table 3).

Discussion
Short and easy-to-administer assessment tools seem necessary to identify children with specific learning disorder who can benefit from early school-based interventions. The PHAT is relatively new and it is the only tool that embodies several variables for identification of handwriting problems. The results of the present study showed that the PHAT has a high to moderate diagnostic accuracy to distinguish between typically developing children and children with specific learning disorder. Discriminative validity and reliability coefficients were also acceptable.
According to the cut-off points and validity parameters obtained for the PHAT in typically developing children versus children with specific learning disorder, formation, spacing, and alignment variables in both domains had significant diagnostic ability but text slant had acceptable accuracy. This result can be attributed to the overall scoring of text slant, for which the entire assignment is scored on a 5-point Likert scale but other legibility variables (formation, spacing, and alignment) scores are a mean score of 12 words in copy and dictation domains. This tool’s ability to accurately discriminate typically developing children from children with specific learning disorder can be useful for researchers and clinicians in future for referral and further investigations. Since no other cut-off values for the PHAT have been reported, the authors are unable to compare the results with other findings. Nevertheless, these results are similar to Koziatek and Powell’s findings on ETCH-Cursive that had a significant (75% total word legibility and 82% total letter legibility) level of accuracy in discriminating between good and poor handwriting for grade 4 children (19). 
Results of the current study on discriminative validity revealed that the PHAT has the ability to separate typically developing children from children with specific learning disorder in both copying and dictation domains. These results are aligned with the study of Barnett et al who found a statistically significant difference (p<0.001) between children with developmental coordination disorder and typically developing children and Havaei et al who found significant difference between typically developing children in grade two and three children (20,25). Large ES was calculated for formation, spacing, and alignment in both domains. Moreover, orthographic error and speed had large ESs. Text slant and size had moderate and small ES in both domains, respectively. Small ES for size in both domains can be due to low impact of word size on general readability of words. Also, most children (typically developing or with specific learning disorder) write words in the same size and only a few children write very small or very big (39). Large ES in legibility components enables clinicians and researchers to use this tool for randomized controlled trials in future.
Internal consistency was acceptable which is in line with previous studies that were conducted in typically developing children (24,26). This result indicates coherence between items in components of legibility that could be explained by the development process of the tool that items were selected based on the opinion of experts. High internal consistencies were reported for Handwriting Proficiency Screening Questionnaire (HPSQ) (α=0.90) and HLS (α=0.92) in typically developing children and children with developmental coordination disorder (20,40). A former study that was conducted in Chinese language with CHAS reported moderate (α=0.65) internal consistency (21). The reason for this moderate internal consistency was explained by the complexity of the nature of handwriting and various skills (motor, sensory, perceptual and cognitive) required to write.
Test-retest reliability, which is used to assess the stability of measurement in different times, showed acceptable reliability. This result was in accordance with the previous study which was conducted in typically developing children (24). Rosenblum also reported excellent test-retest reliability (0.84) which was attributed to the short time interval (one week) between measurements and length of the test (40). On the other hand, Tsang et al reported good to excellent test-retest reliability in handwriting speed, accuracy, and pen pressure (0.72-0.96) and low test-retest reliability in SD of writing time per character (0.10) and SD of size of each character (0.19). Tsang et al explained the results by a long time interval (2 weeks) between test and retest and also fluctuation of handwriting style based on attention, motivation level, and sense of achievement in the writing assignment (21). The values of SEM and MDC in the present study were aligned with former studies (24,26). These findings suggest that this tool has an adequate precision and low error of measurement in typically developing children and children with specific learning disorder.
The present study provides detailed information of handwriting components for clinicians, school-based occupational therapists, and teachers who work with children with specific learning disorder. Researchers can use the cut-off points suggested by the current study to screen the children and implement interventional protocols for children with specific learning disorder, accordingly. Future studies should focus on assessing sensitivity for identifying change over time and determining inter-rater reliability and minimal clinical importance difference for clinical interventional studies.
The authors acknowledge that the present study had limitations. First, environmental noise, low light, and difficulty in determining ergonomic features in some schools were problematic. These features may affect students’ handwriting performance. Second, several skills related to handwriting (e.g., visuomotor) were not measured due to constraints mandated by schools or teachers. Third, different pencil grips may affect control over handwriting. The study would have been more comprehensive if different pencil grips were identified and discriminant validity regarding this matter was examined. Further research is recommended on the raised issues.
 Strengths of this study include careful and random recruitment of children from schools and the application of various measures for determining validity, reliability, and diagnostic accuracy. We suggest future longitudinal studies for determining the sensitivity of the PHAT to change following handwriting interventions.

Conclusion
Psychometric properties of this tool suggest acceptable reliability and validity as well as acceptable to high diagnostic accuracy. Hence, this tool is suitable for screening primary school-aged children with specific learning disorder. 

Acknowledgements
We would like to express our appreciation to the children, their parents, and teachers for their cooperation. The Ethical Committee of Iran University of Medical Sciences granted ethical approval (ethical code: IR.IUMS.FMD.REC1396.9511355012).

Conflict of Interest
Authors do not have interests to declare.