Psychometric Properties of the Persian Version of Student Evaluation of Online Teaching Effectiveness (SEOTE) questionnaire Among Medical Sciences’ Students

Abstract
Background: Due to the increasing number of online educations, traditional methods for evaluation of different on-line education methods are not suitable and there is a need for a valid and reliable tool in this field. This study aimed to examine the validity and reliability of the Student Evaluation of Online Teaching Effectiveness (SEOTE) questionnaire among Medical Sciences’ students in Tabriz, Iran. 
Methods: Using convenience sampling method, the present study included 230 students of Medical Sciences (e.g., mean age of 21.96 ±3.47 years) in Tabriz, Iran. The SEOTE questionnaire was used to collect the data. Forward–backward translation, construct validity and content validity were utilized to check the validity of the questionnaire. In addition, temporal stability was calculated using the test-retest method and Internal Consistency Coefficient (ICC). 
Results: It was confirmed that the SEOTE has appropriate content validity using the content validity index (CVI) of 0.80 and content validity ratio CVR of 0.70. Exploratory factor analysis (EFA) identified the following two factors for domain identification of “communication” and “learning”. A Cronbach’s alpha of 0.97 was obtained for the questionnaire’s reliability, and ICC was used to confirm the temporal stability of the questionnaire (95% Confidence Interval [CI]: 0.966-0.977). 
Conclusion: This study confirms that the Persian version of SEOTE questionnaire is a valid and reliable tool for evaluation of on-line teaching among Medical Sciences’ students.
Keywords: Communication, Confidence intervals, Education, Factor analysis, Iran, Reproducibility of results, Students, Surveys and Questionnaires

Introduction
Using Internet-based online education to deliver courses to university-level students in higher education programs has become a standard method (1). In fact, the use of online education is an accepted strategy to meet the educational needs of individuals in situations where face-to-face education is not possible for any reason (2). These reasons can be the problems of physical presence related to the student himself, the impossibility of providing face-to-face services for the institution, or the occurrence of unforeseeable events (e.g., pandemics, disasters, etc.) at the community level (3). On the other hand, the use of online education is increasing, since many students want to get an educational degree that affects their professional growth and at the same time allows them to have their family and job responsibilities (4). Although offering online courses allows for greater access to educational opportunities, universities and higher education institutions have not paid much attention to the quality of web-based education and have ignored this issue (5). In addition, it is necessary to pay special attention to the importance of education as an important human activity, because investing in proper education is necessary for both professional growth and economic development (6). Therefore, in online education as well, it is a well-known fact that education will never reach its goal without effective facilitators or teachers due to their great role in the responsive education process. This issue is particularly important in online education because the role of facilitator and teacher is very important and the quality of online education is directly associated with their skills in education (7-10). Regarding the effectiveness of online education, several studies have been conducted in various fields of education; in the study of Frazer et al (11), online education was introduced as one of the successful strategies to increase the level of awareness and success of students and to create a spirit of cooperation among them. In general, numerous factors affect the effectiveness of online education including the instructor-student interaction, peer interaction, social media usage, family support, and technical assistance (12). Therefore, it is necessary to provide an appropriate tool to measure the effectiveness of online education. Furthermore, to evaluate the effectiveness of online education in Iran, a valid organized tool (13,14) is needed for continuous evaluation and monitoring of the results. However, based on the literature review, there is no standard questionnaire to evaluate this important outcome in medical education in Iran. Thus, this study aims to investigate the psychometric properties of the Student Evaluation of Online Teaching Effectiveness questionnaire in Medical Sciences’ students of Tabriz University of Medical Sciences.

Materials and Methods
Participants
This cross-sectional psychometric study was conducted among 230 Medical Sciences’ students of Tabriz University of Medical Sciences. The participants were 34.8% males and 65.2% females, with mean age of 21.9±3.2 years old. Data were collected through an online survey by using a convenience sampling method from January to September 2022.

Student Evaluation of Online Teaching Effectiveness (SEOTE)
The original SEOTE was developed by Bangert et al in 2008, and its validity and reliability were approved (15,16). This tool has 26 questions and consists of different parts including: Student Faculty Contact (SFC), Cooperation Among Students (CAS), Active Learning (AL), Prompt Feedback (PF), Time on Task (TT), High Expectations (HE), Diverse Talents and Ways of Learning (DTWL). The SEOTE has a six-point Likert scale (from strongly disagree to strongly agree), along with an open-ended item for expressing students’ opinions and feedback.

Translation
Due to the fact that this questionnaire can be influenced by cultural and environmental factors, the forward-backward Translation method was used (17). Two medical education experts independently translated the English version into Persian, and a third expert was asked to resolve the disagreement. For the survey, an aggregated version was developed. Then, in the next step, the SEOTE was back-translated again to be compared with the original English to ensure the equality of both versions.
Face and content validity
The questionnaire was subjected to a qualitative face validity test. Ten participants were invited to participate in this section. The questionnaire was then evaluated by the respondents. During this process, the scale was modified. An expert panel evaluated SEOTE’s provisional Persian model. In order to ensure relevance and appropriateness, the experts reviewed the items one by one. As demonstrated in other studies (18), Content Validity Index (CVI) was calculated based on three indicators of relevance, simplicity, and clarity. Also, Content Validity Ratio (CVR) was calculated based on ‘necessity’. The wording of each item of the questionnaire was improved based on their suggestions. According to the World Health Organization (WHO) recommendations (19), CVI and CVR of the questionnaire are 0.80 and 0.70, respectively that are acceptable. The face validity of the pre-final questionnaire was checked by a group consisting of 10 students meeting the same study eligibility criteria. The SEOTE in Persian had the same length as the original SEOTE since no question was removed.

Sample size calculation
We used Exploratory Factor Analysis (EFA) to evaluate the construct validity and analyze the data. Based on the correlation between the items, this method requires a sample size to parameter ratio of at least five to one, and ideally ten to one. As the questionnaire contained 26 items, the sample size must be larger than five times of the number of questions. For this study, we considered 230 subjects according to this criterion (20).

Evaluation of construct validity and statistical analysis approach
The construct validity of this study was evaluated using EFA. Kaiser Meyer-Olkin (KMO) and Bartlett’s test of sphericity were used to assess sampling adequacy for factor analysis. To extract the factor, the significance level for each factor, eigenvalues of one or higher were considered. Principal Component Analysis (PCA) with varimax rotation was used whenever the loading criterion was greater than 0.4. Stata Statistical Software (Release 17. StataCorp LLC, College Station, TX) was used to perform confirmatory factor analysis (CFA) to evaluate the fitted EFA model to the observed data. In order to calculate fit indices, the comparative fit index (CFI), Tucker-Lewis Index (TLI), Root Mean Square Error of Approximation (RMSEA), and Standardized Root Mean Square Residual (SRMSR) were used with the following cut-off points: CFI >0.80; TLI >0.80; RMSEA and SRMSR acceptable values from 0 to 1 (21-23). The statistical analysis was performed using SPSS 25 (SPSS 25, IBM Corp., Armonk, New York, USA).
 
Data collection
We collected the demographic variables, including gender, age, and education level of the study participants using appropriate questionnaire.

Results
Construct validity
In the final analysis, we removed the missing data since all items had a missing rate below 5% and the mechanism for missing data was random. Based on KMO of 0.961, the sample size was sufficient for factor analysis (24,25). As indicated by this statistic, weak EFA is equal to values  of less than 0.5, moderate EFA is 0.5-0.7, good EFA is 0.7-0.8, great EFA is 0.8-0.9, and excellent EFA is >0.9 (26). Also, the results of Bartlett’s test were significant (p=0.001) and showed a verifiable relationship between the variables. Due to this, each factor extracted by EFA was related to the others (27). PCA identified two factors for 26 items based on an eigenvalue greater than one. A two-factor solution explained 65.49% of the variance. The scree plot demonstrated a two-factor solution (Figure 1). Based on the rotation factors, loading factors are provided in table 1. The loading factors provide the strength of the correlation coefficient between the item and the identified factor (the item’s priority for the factor). The factors were categorized into two domains of “communication” and “learning”. The CFA was performed on 26 questions of the final questionnaire to determine the fitness of the model derived from the EFA. The fitness of the model is shown in figure 2. Fit indices were calculated using covariance matrices. According to all fit indices, the tests’ result was acceptable. The values of RMSEA and relative chi-square (χ2/df) were 0.106 (90% Confidence Interval [CI] = 0.099-0.113) and 973.94 (p<0.001), respectively. CFI was equal to 0.864 and TLI was equal to 0.851, which are the comparative indicators of the model.

Table 1. Results of Factor Loads for the SEOTE

SEOTE: Student Evaluation of Online Teaching Effectiveness.

* Factor loading higher than 0.4 is acceptable.

Reliability
Cronbach’s alpha was used to measure internal consistency. SEOTE had a Cronbach’s alpha coef-ficient of 0.972, indicating high internal reliability. The values for the subscales of “communication” and “learning” were 0.962 and 0.930, respectively.
To assess the stability of the SEOTE scale, a test-retest analysis was conducted. The results were satisfactory. The Intra-Class Correlation (ICC) was 0.97 (95% CI=0.96–0.97). The ICC for “communication” and “learning” subscales were 0.96 (95% CI = 0.95–0.96), and 0.93 (95% CI=0.91–0.94), respectively.

Demographic findings of the participants
Out of 230 subjects, 80 (34.8%) were men and 150 (65.2%) were women. Their mean age was 21.9± 3.2 years old. Also, 172 (74.8%) participants had a Bachelor’s degree, 50 (21.7%) had a Master’s degree, and 8 (3.5%) had a Ph.D. as shown in table 2.

Table 2. Respondents’ Demographic and Course Charac-teristics (n=230)

Discussion
This study aimed to examine the psychometric properties of the SEOTE. The test-retest method indicated that the questionnaire has good reliability. The Cronbach’s alpha for all the questions in the questionnaire was 0.972, which supports the hypothesis that the variables are related. This scale’s reliability was also confirmed (α=0.95) in USA by Bangert et al (13). Test-retest results (95% CI: 0.96-0.97) over two weeks represented good reliability for the items, which were stable over time. A factor analysis and content analysis were used to assess the structure’s reliability. Using factor analysis, two subscales were identified, with a total variance of 65.49%.
As a result of the cultural differences between American and Iranian people, the present study had fewer factors (two factors) than the original tool, which had four, including ‘Student-Faculty Interaction,’, ‘Active Learning’, ‘Time on Task’, and ‘Cooperation Among Students.’ In this study, the first subscale was ‘Communication’ (with 16 questions), and the second was ‘Learning’ (with 9 questions).
The difference in the number of obtained subscales might be due to the difference in internet accessibility between the people of Iran and other countries (due to limited access of some sites and applications that are an effective tool for interacting with teachers and other students and better learning). Also, Iran’s low fixed Internet speed and relatively higher cost compared to other countries, probably affects the quality of online teaching in Iran. 
This study found that the Persian SEOTE had acceptable validity. As a result, it can be utilized as a valid tool. A CFA model was used to determine whether our hypothesized model fit the data. The EFA’s two-factor model had a good fit, according to the results. 
There are numerous challenges for online education of medical science students in Iran; the most important challenges are technical and technological (weakness in telecommunications’ infrastructure), teachers and learners (unfamiliarity with the structure and technology used for e-learning), ethical challenges (weakness of the existing technologies for fraud detection), and problems related to psychological issues (technology-related anxieties such as power and internet outages and system crashes) (28,29). Also, some of these challenges are issues related to the education, research, and financial issues (30). It seems that student is the most important challenge and in Iran, it is a neglected factor in e-learning; the underlying reason is that in our current e-learning systems, there is a rapid and forced transformation from face-to-face to the electronic methods and it is necessary to take into account any students’ differences to increase the effectiveness of the education and to make a proper environment and educational facilities before any changes from face-to-face education to e-learning (29,31,32). It seems that in medical education, the development of virtual medical simulators and virtual hospitals and virtual cases, promoting telemedicine, and conducting online examinations will be beneficial in promotion of virtual education (30).
The study had some limitations. First of all, no concurrent validity was performed. Consequently, future studies should include this matter in their procedures. Second, different access to the Internet and communication devices such as smartphones and laptops in different areas of Tabriz affects the quality of online education.

Conclusion
This study found that the Persian version of the SEOTE questionnaire had an acceptable content validity and construct validity. It also showed internal consistency and temporal stability, thus future studies on student evaluation of online courses can use it.

Ethical Considerations 
The protocol of the current study has been approved by Research Undersecretary of Smart University of Medical Sciences, Tehran, Iran (Identifier: IR.SMUMS.REC.1402.016).

Conflict of Interest 
None.