Video Conferencing Costs and Benefits: An Evaluation Report

Abstract
Background: Education Development Center (EDC) of the IRAN Ministry of Health conducted a national webinar and evaluated it to know what it achieved, compared to an in-person conference.
Methods: The evaluation was arranged with mixed method design in two quantitative and qualitative parts. In quantitative part, the data was collected through an electronic survey with census sampling. The number and distribution of participants was calculated, the quality of each lecture was rated on a 5-point Likert scale and the cost-effectiveness was estimated by calculating the actual cost. Data collection was done in the qualitative part with formal and informal interviews based on convenience sampling and content analyzed.
Results: High accessibility, synchronous communication, and wide coverage with convenient diversity indicate proper publicity and well acceptance of the seminar. Increasing awareness, inspiring a new attitude, resolving some ambiguities and 82% satisfaction rate show the overall success of the seminar. Temporal constraints, limited interaction, delay in communication or disconnection and ambiguity of sound were things that aroused dissatisfaction of audience. 0.8% cost of the webinar compare with a hypothetical in-person conference shows acceptable cost -effectiveness. 
Conclusion: Both in the corona virus pandemic or the post-pandemic era, if the purpose of the educational program is to inform, increasing awareness and motivate the audience, conducting such a national webinar is optimal and recommended and increase equitable access to national experts.
Keywords: Cost-effectiveness, Communication, Survey

Introduction
Video conferencing became a compulsion in the pandemic covid-19, but was not new. Video conferencing has been used since 1968 (1) and was the growing mediums in distance education (2-4) and a potential alternative for teacher-learner interaction (5,6), even before the COVID-19 pandemic. UNESCO has estimated that 91.3% of the total enrolled students in schools across 188 countries which had been affected by the COVID-19 pandemic until April 2020 (7), turned to teaching online, particularly video conferencing (8). With the decline of the Covid pandemic, UNICEF welcomed the reopening of schools (9). Paul Trac believes that the biggest problem of virtual classes is the reduction of training multi-sensory communication feedback loops (10) which can be one of the reasons to emphasize face-to-face training. From now on and again, the choice between in-person training or online training is in front of educational professionals that strong reason should support this choice.
Teachers are the main actors in any educational scene and in the new world, faculty development is a very important educational event to ensure that their students become competent human capital. Even before the coronavirus pandemic, time constraints and other competitive conditions in staff training, such as high travel costs, the scarce accessibility and the high wages of experts, had established a new urge in organizations to imply new forms of virtual technology (11). In this study, a national faculty development event in the form of video conference is compared with a hypothetical face-to-face conference to reveal the advantages and disadvantages of such programs from various perspectives. A realistic look at this phenomenon at that time (before the pandemic) can reveal strengths and weaknesses in a more realistic situation without bias of the impact of the Corona pandemic.
Education Development Center (EDC) of Iran Ministry of Health and Medical Education (MOHME) is empowering the faculty members of universities (University of Medical Sciences or UoMS) in different areas following its missions. National coverage and more participants are other goals of these educational programs. These goals along with technological advancements in e-learning and the necessity of cost savings due to economic sanctions were reasons for moving to implementing the faculty development in the form of webinars before the coronavirus pandemic. 
A nationwide seminar titled “Educational Scholarship in the looking glass of Faculty Promotion Regulations” was held on July 16, 2019 and 51 gathering halls of UoMSs were connected to the presentation center through video conferencing. This was done in cooperation with the Iranian Shooka Company. The UoMSs widely publicized the seminar through printing posters which were customized according to the specific conditions of each university. The educational goals of the seminar were to inform and increase the awareness of faculty members, their Board of Auditors, and the executive members of scholarship committees regarding the new revisions in the scholarship promotion regulations. Four lectures were planned and conducted on “The basics of scholarships of teaching and learning (SOTL)”, “Evaluation of SOTL”, “Strategies for promoting SOTL”, and “SOTL in light of the promotion regulations”. For better communication during the Q&A sessions, a virtual group was formed for each lecture with the university representatives on the WhatsApp social network, which facilitated the collection of questions related to each lecture. Immediately after the end of each lecture, questions were answered. The recorded lectures had been sent to the universities for offline broadcast as an alternative method in case of any technical communicative failure.

Materials and Methods
The study is a program evaluation with mixed method design conducted in two quantitative and qualitative parts. The proposed design of the study was reviewed in EDC of MOHME and after approval in terms of evaluation criteria, method and compliance with ethical considerations including freedom of participation, respecting confidentiality principles and balance reporting, it was implemented.

Quantitative evaluation
In this part, information was collected in three sections: 
1. Calculation of the number and distribution of the participants based on quantitative statistics received from the UoMSs.
2. Evaluation of the quality of each lecture and the seminar by the participants. Sampling was census. The data were collected through an anonymous electronic questionnaire, and the participants’ overall impression of each lecture was rated on a 5-point Likert scale from poor to excellent. It is necessary to be mentioned that the large number of participants, the lack of forcing to participate in the entire session and the distribution in 51 universities, greatly limited the methods of studying the educational outcomes. Therefore, participants’ satisfaction as first level of educational outcome of Kirkpatrick pyramid was considered as the evaluable education outcome. Of course, what was related to achievement of the goals in the collected texts was analyzed in the qualitative part.
3. The cost-effectiveness was estimated by calculating the actual cost and comparing it with the hypothetical holding of such a conference in Tehran’s Razi International Conference Center, which is one of the largest centers for holding such scientific conferences.

Qualitative evaluation
Data collection in the qualitative part was done manually by the EDC of the UoMSs with formal and informal interviews. Based on convenience sampling, a number of participants were asked to escribe the strengths and weaknesses of each lecture and the seminar as a whole. The collected text by the repersentatives of UoMSs were analyzed and categorized based on content analysis. All findings, whether positive or negative, regardless of the number, were included in the final conclusion to obtain a balanced and complete report. 

Results
36 universities from the 51 universities (70% response rate) sent their collected data which is reported as follows:
Number and participant composition: 1723 members of the Board of Auditors, educational managers, faculty members, students, and educational experts (from 36 UoMSs) participated in the seminar (Table 1).

Table 1. Distribution of participants in the seminar “Educational scholarship in light of faculty promotion regulations” (N = 1723) (2019)

* Note: The composition of 62 participants of the X UoMS was unknown and not included in the categorization.

36 UoMSs sent their data. Shahid Beheshti UoMS (149 participants), Ahwaz Jundishapur UoMS (110 participants), and Khorasan Shomali UoMS (109 participants) had the most participation with more than 100 participants, and Asadabad UoMS (9 participants) and Iran UoMS (11 participants) had the least participation.
Quality evaluation of lectures: 317 participants responded to the e-questionnaire (about 18% of the response rate). However, they did not answer all the questions. In general, 82.3% of the participants described the quality of the seminar lectures as good and excellent, and 2.6% rated it as poor and very poor (Table 2).

Table 2. Evaluation of the quality of 4 lectures by the participants of the seminar “Educational scholarship in light of the faculty promotion regulations” (N = 281-309) (2019)

*Note: For confidentiality, the order of the table is not based on the order of lectures.

Cost-effectiveness evaluation: The executive cost of this program for the Deputy of Education of the Ministry of Health and Medical Education was 105/000/000 Rials as shown in table 3.

Table 3. The real cost of the seminar “Educational scholarship in light of faculty promotion regulations” (2019)

It should be noted that the cost of holding by the MOHME was lower than the cost of outsourcing it to the private sector. Moreover, the cost of conducting this program in the UoMSs using the existing hall and infrastructure, without the need to establish new structures, was limited to provision of snacks and without any other costs for them. If such a seminar was held in person and under the same conditions in Tehran, the total cost would have been 12,407,000,000 Rials according to table 4.

Table 4. The cost of hypothetical holding of an in-person seminar “Educational scholarship in light of faculty promotion regulations” in Tehran (2019)

If the cost of accommodation and lunch for these participants were not calculated based on the current contractionary policies, the minimum cost would be about 7.400 billion Rials. Therefore, the real cost to hypothetical cost ratio is 0.8%, which designates the optimal performance of this seminar.

Qualitative evaluation
The specific evaluation of each lecture was reported to the lecturers and due to confidentiality, they will not be mentioned. In the general part of the qualitative evaluation, the strengths and weaknesses and suggestions were analysed (content analysis), and the results were classified into the following 5 categories (Table 5).
Preparation and planning (56 repetitions)
Content and content organization (45 repetitions)
Conducting seminar in the form of a video conference (40 repetitions)
Conference lectures (24 repetitions)
Goal attainment from the viewpoint of the participants (15 repetitions)

Table 5. Strengths and weaknesses of the seminar “Educational scholarship in light of faculty promotion Regulations» in Tehran (2019)

* “SOTL in light of promotion regulations” The topic was mentioned 4 times in the contexts that it should have been longer, and one person mentioned that it could be removed.

** “Strategies for promoting educational scholarship” The topic was perceived as not practical. It also does not match the audience (suitable for managers only).

*** “Evaluation of SOTL” The topic was suggested to be a specific workshop.

Discussion
High accessibility, synchronous communication, wide coverage (51 universities at once) and high participation (1723 participants in the lowest estimate) with appropriate diversity (18% managers and members of the Board of Auditors) shows the adequate publicity and audience interest which with 82% satisfaction rate, indicates the overall success of this seminar.
The imposed cost compared to the actual cost of the hypothetical face-to-face conference (0.8%) confirms the optimal implementation of this seminar. It should be noted that the reduction in the indirect costs, including the time saving of the participants as well as the environmental benefits resulting from the reduction of inter and intra-city transfers, were not included in the cost-effectiveness analysis. This number of participants have not been seen before even at the most well-attended annual Health Profession Education conferences. Also, the largest existing conference hall, the Razi conference, with the capacity of 1000 spectators, could not accommodate such a crowd at once. The general look at the results along with the reduction of potential transportation risks across the country shows the cost-effectiveness of holding such conferences.
In the qualitative part of evaluation, the dominant image of the seminar can be described as follows:
Useful and practical content, accurate time management, discipline in implementation, and em-ploying outstanding lecturers increasesed awareness, inspiried a new attitude and resolved some ambiguities associated with the content. At the same time, temporal constraints, limited number of examples in the lectures, not having answered all the questions, limited interaction, delay in communication or disconnection and ambiguity of sound were things that aroused dissatisfaction of the audience. However, proper interaction in new condition requires more planning. The social networking groups in this seminar somewhat connected the lecturers to the audiences, but overall, it was not satisfactory and should be more planned.
The Shooka software was not up-to-date in most of the UoMss and half an hour before the start of the seminar, created a heavy workload, which could have been easily resolved if prior arrangements had been made. More training of technical supporters, constant communication, up-to-date platforms, and promptness for resolving unexpected issues by the sponsoring company can undoubtedly improve the quality of such webinars.

Conclusion
The study shows that if the purpose of the educational program is to inform, increasing the awareness and motivating the audience, holding this type of seminar, especially national webinars, would be optimal even after the COVID-19 pandemic. Arias et al demonstrated that the learning outcomes of video conferencing are no different (neither better nor worse) than traditional face-to-face classes, but course objectives may play an important role in determining the effectiveness of alternative delivery methods (12).
The Shooka system in this study could be replaced with other softwares to allow each user to connect separately, that facilitates the connection of the faculty members at any point. In individual synchronous videoconferencing, participants can see each other and contribute in spontaneous interactions, immediate feedbacks, and social presence via audio, video, or text (5,13), but compared to this seminar, social interaction probably reduces in each UoMs. Video conferencing is an alternative method for teacher-learner interaction (5,6) and has enormous potential to enrich distance learning (14).

Acknowledgements 
The authors of the manuscript would like to thank all the faculty members of Universities of Medical Sciences which cooperated in collecting the data.