Evaluation Report on Vyew Web Conferencing as an Instructional tool

EDTECH 505 Final Evaluation Report

7/25/2012 Submitted to: Northern Beacon School Submitted by: Fiona Springer Principal Evaluator

Silhouette Evaluations
Stand Out from the Crowd

John Thompson Boise State University Summer 2012

Summary Vyew is a free browser based web conferencing service that enables online video, audio, and text based collaboration and supports sharing of content in a range of commonly used formats. Features enable users access to content, which can be reviewed, and commented on during or after a conference occurs. This report evaluates Vyews perceived merit as an instructional tool from the teachers perspective in the areas of curriculum support, global participation, student-directed learning opportunities, and enhancement in teachers technology skills. The data were gathered from unstructured interviews with selected curriculum leaders in the school, and from a crosscurricular sample of teachers exploring basic features in Vyew, engaging in a Vyew web conference, and completing a survey on their experience and perceptions of the program. The report found that the presentation of curriculum content could be extended and enriched through an array of sources supported in Vyew. Differentiated learning and student directed learning activities are facilitated by having live and asynchronous access to content. By enabling remote connections, teachers can explore cross-national participation. Teachers skills and knowledge of communication technologies can be facilitated through exposure to those features in Vyew. The report further recommends an experimental implementation of Vyew after additional assessment of technical and logistical issues and conducting training in a web conference setting. Description of the program evaluated Vyew is a web conferencing service that allows online collaboration and sharing of content in a synchronous and asynchronous way and as such may have potential merit as an instructional tool in the Northern Beacon secondary school. It is not currently in use at there, but the school introduces technology options to staff as part of its instructional drive. This occurs at different times in the school year with at least one option being initiated usually at the start of the academic year. Teachers therefore examine and make decisions on which technology options may usefully be incorporated in their classrooms. Over the period of a month, this program evaluation focused on the teacher perspective to determine the extent to which Vyew could meet instructional needs in a Barbadian secondary school. Program Objectives After conducting an assessment of teacher needs and examining program capabilities of Vyew, four evaluation goals were identified. The evaluation goals and objectives that were used to measure the merit of Vyew as an instructional tool in Northern Beacon School were: 1. Vyew web conferencing can enhance curriculum in various subjects. i. All teachers will identify at least one way they can use Vyew to present curriculum content. 2. Teachers can use Vyew to provide opportunities for global participation. i. All teachers will successfully connect remotely to other persons. ii. Seventy five percent of teachers will identify potential resource persons for future conferencing. 3. Teachers can use Vyew to provide opportunities for students to direct their own learning.

i. All teachers will identify at least one way Vyew can enable student-directed learning in their subject. 4. Teachers can enhance their communication technology skills and knowledge. i. Ninety percent of teachers will self-assess improved mastery of communication technology features in Vyew. ii. Teachers will self-assess the likelihood of applying communications technology skills and knowledge in their teaching and personal life. Program Components Vyew is a free browser based web conferencing service, therefore software does not need to be installed. It operates on PC, Mac, and Linux platforms and allows up to 10 participants at one time. Paid versions extend the number of participants by 5 persons. Participants meet in rooms to which a range of content can be uploaded. Supported content includes Microsoft Office files, PDFs, text files (.txt, .rtf), images (.jpg, .png, .gif, .bmp), Flash, audio and video (.swf, .mp3, .flv). The administrator can manage access rights and user roles in the rooms. Other notable features include desktop sharing and screen capture. Participants can access content synchronously and asynchronously, so someone absent from an online meeting can view, edit, and comment on saved content. Multilingual support is provided. A number of communications and annotating tools are available in Vyew: Voice Over Internet Protocol (VOIP) talking to others using a computer microphone and headset; Webcam- a maximum of 4 participants broadcasting video to the others; Teleconferencing via a standard phone; Text chat with all viewers or privately with a single participant; Voice notes- leaving voice mail comments; Sticky notes- embedded comments anywhere on the page; Whiteboard and drawing tools to add to content. Evaluation Method Participants The Northern Beacon has nine subject departments and a staff complement of 52 teachers who are mainly trained graduates or trained, special grade teachers. Since the program evaluation occurred at the end of the school year and into the summer break, it was impossible to use the entire population as recommended by Boulmetis and Dutwin (2011) where numbers are under 100. Instead, using stratified random sampling, 18 teachers were identified with 2 persons being selected from each Department. However, from the 18 selected only 11 actually completed the survey on which the evaluation was based. Heads of Department (HODs) and the Deputy Principal are curriculum leaders in the school providing guidance to teachers in a number of areas, advising on instructional tools and strategies, and supervising teachers in the execution of their duties. Three 3 HODs were sources of information on curricular requirements.

Procedures To evaluate Vyew as an instructional tool, the goal-based model of objective attainment was used. Data were collected and analyzed to measure the extent to which the objectives were achieved. The program began with an examination of existing documents that guide instruction in the school such as the schools mission and vision statements and syllabi. In unstructured interviews, Department Heads identified requirements in Caribbean Examinations Council (CXC) Caribbean Secondary Education Certificate (CSEC) syllabi in Foreign Languages and General Studies which helped to clarify the needs on which the objectives were based. Within a one week timeframe, teachers independently completed Vyews Getting Started Tutorial introducing them to the basic program features. A time was set for a synchronous meeting in a designated room with the evaluator. Those unable to meet at that time were invited to visit the meeting room at their convenience with the possibility of engaging if the evaluator was online at that time. Lastly, teachers completed an online survey which gathered their perceptions of Vyew for instructional purposes. Invitations to complete the tutorials and survey were sent via email. (Appendix A) The original intent in the evaluation program design was to engage all participants during school time in a synchronous web conference to discuss the merit of Vyew for instructional purposes. However, the schools access to Vyew was not unblocked by the Ministry of Educations servers before school broke for summer vacation. Due to prior commitments, it was impossible to coordinate a time for all, resulting in the procedures already described. Only 3 teachers attended the synchronous conference, and 11 of the original 18 completed the survey. Data Sources The main data source for the four objectives was the online survey, which was informed by the results of the unstructured interviews and review of existing documents described below. Unstructured interviews were based around 3 open ended questions related to personal comfort with and use of technology in education; teachers challenges in content delivery; and their preference for constructivist or teacher centered classrooms. Follow up questions were asked depending on the responses given. This technique was chosen to encourage a freer flow of ideas and perceptions. The interview questions are referenced in Appendix B. Existing Documents examined were the mission and vision statements of the school to learn the philosophy guiding the schools culture. The rationales and content of Social Studies and Foreign Language syllabi made references to technology inclusion, global participation, and student directed learning. Excerpts from these statements are seen in Appendix C A Survey was created using Google Docs to gather teachers perceptions. It comprised demographic information such as the number of years and areas taught, and qualifications. There nine questions divided across three sections called Instructional Merit, Vyew Features and Tools, and Personal Satisfaction/Experience. Each section contained at least one 5-scale item capturing opinions or levels of competence. Open ended questions were also included to gather fuller responses. Qualitative data was mainly derived from the items. The Evaluating Vyew Web Conferencing survey can be accessed by the hyperlink. Screenshots are also in Appendix D

Results This section reports on the results of the online survey in relation to the program objectives. Teachers completed the online survey after working through the Vyews Getting Started Tutorial and in some cases, engaging in a synchronous Vyew meeting with other respondents and the evaluator. Objective 1: Vyew Web Conferencing enhances presentation of content in several subject areas. Figure 1

Present and Enhance Content with Vyew

No. of responses 8 6 4 2 0

Present content Present creatively Use resource persons

Degree of Response

All 11 respondents believed they could present content in their respective subjects using Vyew, with 2 persons unsure if they could use it to present in a creative manner and one person unsure about identifying resource persons in their content area. In the open response question, all were able to outline activities for their classroom that could be taught using Vyew. Examples included: remote sharing of acting/actor tips with students in Theatre Arts; virtual field trips in Geography, foreign languages; using 3D content, video to present processes/events e.g. planetary rotation, internal combustion engine; using images to encourage shyer children to open up and communicate more freely

Objective 2: Teachers opinions on making remote connections using Vyew and how these could lead to global participation. Teachers looked at if Vyew enabled them to establish links to others within the school, in other schools and countries, and identify potential resource personnel for conferencing. In figure 2 below, most respondents agreed they could make connections, with 10 teachers positive they could identify resource persons, 9 connecting within the school, and 8 making connections in other schools and countries. There were 5 uncertain responses overall and 1 response where the teacher disagreed about making connection within the school.
5

Figure 2

Enable Remote Connections

10 No. of Responses 8 6 4 2 0

Within school
Other sch/countries Use resource persons

Degree of Response

Objective 3: To determine if teachers thought Vyew could help students direct their own learning. Teachers responded to the question directly and supplemented with examples in the open response section. Figure 3 below shows 73% or eight persons felt students would be able to direct their own learning using features in Vyew, with 2 being uncertain and 1 person disagreeing that it could happen. The respondent who disagreed related this to poor levels of motivation among her current batch of students. Figure 3

Students Direct own Learning Disagree

9% S Agree 18%

Uncertai n 18%

Open responses for objective 3 tended to fall into categories such as providing for differentiated learning activities or tutorials where students could proceed at their own pace; interactive sessions where student got feedback from each other or worked together to create or complete assignments; mentoring from either stronger peers or from experts in the field who would provide critiques of their independent efforts.

Agree 55%

Objective 4: Can Vyew could enhance teacher skills and knowledge of communication technology. Table1
VOIP

Competence with Communications Tools

Webcam Broadcast Pre Post Pre Post Text Voice Sticky TeleChat Note Note conferencing Pre Post Pre Post Pre Post Pre Post 2 1 2 1 5 3 2 3 6 1 6 3 3 8 5 2 5 5 5 1 3 7 2 1 2 3 1 4 2 2 3 5 6

No Knowledge Awareness Guided Use Independent Use Mastery

2 1

3 5

3 6

Table 1 shows how teachers self-assessed competence in the six communications tools before and after exploring Vyew. In all instances, there was improvement in ratings of knowledge and skill. Text chat, sticky note, and teleconferencing achieved the levels of independent use and mastery after using the program. Only in one instance did a respondent post-assess competence at awareness using voice note, but was that due to microphone issues hampering experimenting with this feature. Item 7 on the survey asked teachers to rate some statements according to their personal feelings and experiences with the program. In analysis, the statements were classified as either positive or negative reactions to Vyew. From a total of 77 possible responses all negative reactions accounted for 3, with all uncertain reactions, and all positive reaction accounting for 6 and 68 respectively. The 3 negative responses and 4 of uncertain responses were from teachers who did not think they would use the technology in their personal life. However, all felt they would use it in teaching, with 10 agreeing it aligned with their teaching style. Most did not see training and technical issues as problems to overcome. Discussion The purpose of this evaluation was to determine if teachers perceived instructional merit in Vyew Web Conferencing service and ultimately if it should be introduced as an instructional tool for the upcoming school year. The areas of instructional merit were limited to the 4 objectives based on teacher needs and program capabilities. The results seem to indicate that Vyew met the objectives and could be a worthwhile addition as an instructional tool. Objective one: To determine if Vyew enhances the presentation of content in a range of subject areas. i. All teachers will identify at least one activity to present content. Given the range of commonly used file types that can be uploaded in Vyew, teachers will have many more resource materials at their disposal to add richness to their lessons. Instead of only
7

textbooks, or traditional images Vyews platform facilitates combining sound, text, audio, video, and resource people into a lesson. Plug-ins, such as the graphing and equations calculators, the equation editor, and the polls/quiz creator extend its functionality and ability to create content prior to and during a conferencing session. As teachers become more familiar with Vyew and share ideas, it is likely that more creativity will occur in their instruction. In HOD interviews, making lessons appealing for young people who are so technology oriented was seen as a challenge. When asked what they most enjoyed about Vyew teachers reiterated comments such as present science in a fun, interactive way that [is] appealing and engaging, and combine text with visuals to provide an educational, yet fun tool for interactive learning. Given the feedback from the interviews and survey responses, Vyew meets this objective. Objective two: To determine if Vyew enables global participation. i. All teachers will successfully connect remotely to other persons ii. Seventy five percent of teachers will identify potential resource persons for future conferencing. Measuring this objective depended greatly on having a synchronous meeting with the participants in the evaluation. Only 3 teachers were able to attend the live conference and a further 2 met with the evaluator in individual online meetings. The other 6 survey respondents only worked through the tutorial but did not make remote connections during the actual program evaluation. It is probable that the respondents who were uncertain or disagreed about making connections within and outside the school/country might have been among the latter group. The question of global participation was not adequately captured in the questionnaire design. Items asked about making remote connections but did not address for example, how teachers might structure transnational joint projects. Nevertheless, the very nature of web conferencing makes global participation possible once the teacher can organize contacts. Due to these limitations in the design and implementation of the evaluation, while it can be inferred that Vyew can satisfy the objective of global participation, the results can only conclude that it will allow remote connection. Objective three: To determine if teachers can use Vyew to provide their students opportunities to direct their own learning. i. All teachers will identify at least one way Vyew can enable student-directed learning in their subject. Respondents generally thought that Vyew could facilitate student-directed learning. Features include being able to have more than one meeting room, to which specific students could be assigned to engage in differentiated learning activities at a pace comfortable to them. Furthermore, content in the rooms can be saved, accessed, reviewed, and commented on both synchronously and asynchronously, offering students a way to assume responsibility for their learning and understanding. Teachers thought interactivity would promote mentorship where more gifted students could assist their peers, or where students could interact with resource persons asking questions in areas of interest to them. Allowing commenting and using whiteboard features on lesson content would help students connect more closely with the content and with each other instead of being passive receivers in a teacher centered learning environment. Therefore, it would appear that Vyew meets this objective.

Objective four: To determine whether Vyew enhances teacher skills and knowledge of communications technology. i. Ninety percent of teachers will self-assess improved mastery of communication technology features in Vyew. ii. Seventy five percent of teachers will self-assess the likelihood of applying communication technology skills and knowledge in teaching and or personal life. Teacher self-assessment in relation to competence in the technology tools may be open to concerns about the accuracy of the rating. However, since the teachers actually were able to use the tools during the tutorial, it is reasonable to assume there was indeed improvement. Of note, even before using Vyew, 43 out of 66 or 65% of total responses rated competence at independent use and mastery. It would not be unsurprising that they had enough competence to work successfully through the tutorial then to achieve a result of 91% or 60 out of 66 responses after using Vyew. Having or seeing improvement in skill may give teachers confidence to use Vyew, suggesting to them that the program is relatively easy to use. It may be the reason why only one person each thought training in Vyew was necessary or was uncertain if it was needed. Therefore, the evaluator believes the program satisfies this objective. Based on the data collected, there is instructional merit in introducing Vyew in the Northern Beacon School. These are the recommendations of this evaluator: 1. Since the program focused only on the teacher perspective, some assessment must be done on the technical and logistical requirements of implementing Vyew in the school. Consideration should be given for example to bandwidth and location to conduct video conferencing. 2. Although teachers did not feel the need for it, training sessions should be organized to explore more fully the functionality of the program. This is especially critical since all respondents did not experience conferencing during the evaluation. In the sessions teachers will need to create and host meeting rooms which will be required in their classrooms. 3. The training conferences should also be used to further develop discussion in practical ways to use Vyew for content presentation, global participation, student directed learning, professional development, and other areas of educational need. Readings on literature is recommended. 4. Vyew could be implemented on an experimental basis in a department or subject area that is willing. The implementation should be planned with a structured program evaluation component to provide formative evaluation for corrective purposes during the school year. A summative evaluation at the end of the year should be conducted to assess if the perceived benefits have actually been achieved; to see if there have been changes in curriculum delivery; if global/regional interactions have occurred through e.g. establishment of joint projects; if teachers have shared knowledge and skills, which conferencing features were or were not used; impacts on students; and obstacles to the implementation of Vyew web conferencing service at Northern Beacon School.

Silhouette Evaluations
Stand Out from the Crowd

INVOICE
INVOICE #249 DATE: 7/27/2012

Rocky Shores, St. Lucy Phone 246 444-8888 Fax: 246 444-9999
TO: FOR:

Secretary Treasurer Northern Beacon School Fort Orange Speightstown, St. Peter (246)444-5555

Evaluating Vyew Web Conferencing Service

DESCRIPTION

HOURS/DAYS

RATE

AMOUNT

Evaluators services: Creating survey; Vyew Writing Room; Data Analysis; Report writing. Operational Expenses Evaluation Report* electronic distribution

20 days

$500 p/day

$10,000.00 300.00 $00.00 $10,300.00

TOTAL
Make all checks payable to Silhouette Evaluations Total due in 15 days. Overdue accounts subject to a service charge of 1% per month.

THANK YOU FOR YOUR BUSINESS!

The main cost of the evaluation is for services rendered by the evaluator. The cost include reviewing Vyew program capabilities, creating data collection instruments (interview questions and survey), creating a meeting room for a synchronous session in Vyew, data analysis, and preparation of the final report. The total time spent is calculated at 20 days at rate of $500 per day. Operational expenses cover office supplies and utilities. In order to contain expenses, Silhouette Evaluations will disseminate copies of the final report electronically.

10

Appendix A Emails inviting participation in the evaluation and survey. 27 June Hello Colleagues, I would be grateful if you could help me with a project I'm working on for my Masters in Educational Technology. I want to evaluate a Web Conferencing tool called Vyew. At this time, I'm only looking at it from the point of view of teachers determining if there's merit in it for educational purposes in (our) school. I'd like you to go to http://vyew.com/room#/95665858/Getting_Started_Tutorial and get familiar with the features discussed in the tutorial. Just follow tutorial instructions. You don't have to create your own meeting at the end if you don't want to. Tell me your impressions/ideas about it later. The good and bad impressions are needed. Once you get familiar, please join me at http://vyew.com/room#138947/Writing_Room on Monday 2 July between 10:00 and 11:00 am for a web conference discussion if this time is convenient to you. The password is bajanbustop (Note only one 's'). You will see me as fabspring. Once together, we can share ideas there and then! Isn't that great! BTW, it cannot be accessed at school right now, because the Ministry has not yet unblocked the site, even though they promised to. If you have the time to do this (especially at this time of the school year) I'd be ever so grateful. Thanks so much Fiona Springer.

Follow up email July 3 Hello again colleagues, Many of you were unable to meet up yesterday for our session, but I'm still asking for your help in evaluating Vyew web conferencing tool for educational purposes. I'd really like you to check it out at http://vyew.com/room#/95665858/Getting_Started_Tutorial . As you play with the features, could you ask yourself these questions? How can I use web conferencing to present/enhance teaching in my subject?
11

Do I know potential resource persons I could connect with so they could conference/share with my students? How could I use Vyew to let my students direct some of their own learning experiences? Remember you can also go to http://vyew.com/room#138947/Writing_Room where you may see me online as fabspring. The password is bajanbustop If we are online together, we can have a real conferencing experience. I would like you to respond to this survey questionnaire at this link: https://docs.google.com/a/u.boisestate.edu/spreadsheet/viewform?formkey=dFRj TDd6RG9KZ203SG1CNTdtOHFxWXc6MQ The questionnaire shouldn't take more than 5 minutes to complete. I would really appreciate your questionnaire responses by Mon16 July, so I can complete my analysis and write up by 20 July. Thanks again. Fiona Springer.

12

Appendix B
Interview Questions for Heads of Department to identify curricular needs and determine program objectives. Personal comfort with educational technology 1. How comfortable are you with introducing educational technology in your department? 2. How comfortable are you with using educational technology in your classes? 3. What technologies have you found most useful in enhancing content in your subject? Challenges to content delivery 1. What do you see as significant challenges in content delivery in your department? (student motivation, cost of materials, additional resources) 2. Would remote access to people or resources be a plus for your area? 3. Are there aspects of syllabus requirements you find challenging to deliver? Constructivist or teacher centered teaching 1. Do you tend to favor teacher centered lessons or student centered lessons? 2. Do you think teachers in your department are generally constructivist or are their methods more teacher centered?

13

Appendix C
Northern Beacon School Mission Statement excerpt: to provide a challenging and supportive environment through excellent teaching and coordinated exposure, ensuring students explore and utilize knowledge, develop positive attitudes, and attain measurable success, and are responsible, creative and productive in an ever changing, technology driven, global society

Northern Beacon Vision Statement excerpts: Members display mastery of technology skills and use them for problem solving and knowledge utilization, manifest creative thinking and a willingness to explore and innovate, are qualified, trained and professionally supportive

Excerpts from CXC CSEC Social Studies syllabus On completion of this syllabus, students should: 1. understand himself or herself as a social being, his or her relationships with his or her family, community, nation or state, the Caribbean region and the wider world; 4. use skills for social participation and social action; 5. appreciate the human and natural diversity of the region and how this can be harnessed to promote its sustainable development through regional integration; 6. become aware of the global issues and institutions which influence his or her life and the development of the region.

14

Appendix D
Screenshots of Online Survey. Or Click on the link https://bitly.com/NYJA2V+

15

16

17

18

Appendix E Tagxedo cloud of the open responses in the survey.