Evaluation Report: Meeting with Stakeholders

An initial meeting was held with stakeholders involved in the implementation of a new software at our school. We collaborated with peers and subject matter experts to analyze the software and design an evaluation plan to see if this software was meeting our desired outcomes. Here is a summary of that meeting.

Meeting With Stakeholders – Word Document

 

Baughman

Meeting with stakeholders

I am evaluating the efficiency and effectiveness of a program chosen to reduce the amount of grading time teachers spend on providing effective feedback to students on student created language videos.

Leg A = Leg A is a needs analysis designed to identify where stakeholders are currently at in regards to the program. For this leg we had a group discussion with teachers and stakeholders to identify how much time they (teachers) were spending on grading student created videos and providing feedback to students. We addressed how they currently do it and why. We also discussed the effectiveness of this feedback and how teachers desired for their feedback to be more effective in reducing the amount of student repeat in error.

Leg B = Leg B is the planned “bridge” on how to get from Leg A to Leg C. In order to reduce grading time and provide more effective feedback for students (or reduce student repeat in error) stakeholders decided to purchase GoReact, a contextual video feedback tool. Other “bridges” were discussed including a training seminar, the use of free software, and in class assignments instead of student created videos.

Leg C = Leg C is the future goals of the stakeholders. Before beginning the program the evaluator sat down and discussed with stakeholders what their goals and objectives were for the coming semester. The following goals were outlined:

  • Reduced grading time for teachers (efficiency)
  • More effective feedback for students / decrease in repeat student error (effectiveness)

Addressing figure 3.1 Program Cycle in terms of the program I plan to evaluate, The two goals have been outlined by program stakeholders: reduced grading time for teachers (efficiency) and more effective feedback for students / decrease in repeat student error (effectiveness). These were identified by doing a needs analysis of the students and teachers involved in the program. The needs analysis included a group discussion of the current issues ASL teachers a facing. During the program planning phase, the evaluator was present in order to better plan an evaluation of the avenue chosen to address the outlined goals. The chosen avenue was the purchase and use of the contextual video feedback tool GoReact. For the purpose of formative evaluation, the evaluator will have access to data analytics through the software as well as the ability to observe teachers and student using the software. A summative evaluation will be given at the end of the cycle (one semester) in the form of a survey developed by the evaluator.

 

Evaluation Report

This was a semester long project where we evaluated an educational technology. I chose a technology that was newly being implemented in the ASL department at BSU. I followed ethical considerations when conducting research on real people while still applying formal inquiry strategies. It was an eye opening experience to reflect on the effectiveness of technology supported instruction.

Evaluation Report – Word Document

The below embedded document is missing graphics not supported by this WordPress theme. To see the document in full please download it above.
Screen Shot 2016-10-17 at 2.15.47 PM.png

SUMMARY

The use of video in the classroom is thought by many to have a major impact on education, but this impact has yet to be felt due to factors, including but not limited to, video assignments lacking the functionality to provide students with immediate and effective feedback. The ever pressing issue of collecting and providing feedback on instructional and student created videos has drawn attention to the lack of video assessment software available to teachers. This issue can be especially pressing in American Sign Language classrooms as the nature of the language tends to require use of multiple student created video assignments. One common method of assessing student created videos is to have students upload their video to YouTube where teachers review and grade it online, then provide written feedback separate from the video itself. Providing effective feedback is difficult enough, while providing effective feedback on student created videos is nearly impossible. Trying to write out feedback on a rubric or in an email is lengthy and ineffective as students rarely go back and watch their video while reviewing written comments. Also, the use of written English to provide feedback on American Sign Language use and skill directly conflicts with full-immersion theories of language learning. According to these theories, the native user (teacher) should provide feedback in the language being learned to reinforce the acquisition process. The lack of ability to provide this could be detrimental to the progression of student learning as the significance of effective feedback has been noted in multiple  (Blair et al, 2013; Eksi, 2012; Rodgers, 2006; See, Gorard, & Siddiqui, 2015). One emerging technology, GoReact, has been developed to alleviate this feedback issue. However, few formal evaluations on its effectiveness have been done.

 

An American Sign Language (ASL) department at a mid sized northwest university will begin employing the use of GoReact, a contextual video feedback tool, with two courses in the Spring 2016 semester. GoReact is a cloud-based software that allows students to create and upload videos for instructors and peers to access in order to provide immediate contextual feedback into the video via audio, video, or written comments. Prior to the use of this software teachers were reviewing student videos submitted on YouTube and providing written feedback via email. The department requested an evaluation of GoReact in order to decide whether or not to continue its use with the entire department. The usability, efficiency, effectiveness, and impact of the tool was evaluated to measure progress toward specific administrative and organizational program objectives.

 

This report is written to provide tangible data, analysis, and evaluation of GoReact in a real case-study environment. Initial meetings were held with stakeholders to determine objectives at administrative and organizational levels. Those objectives were developed into survey questions and distributed to all students and teachers using the GoReact software at this mid-sized northwest university. Participants included two teachers and two classes of 25 students each. Surveys collected a range of data from Likert scale questions in regard to usability with certain functions to questions addressing personal experiences with the software. The data was then analyzed and evaluated in terms of identified objectives and discussed with regards to which objectives were reached, which were not, and possible improvements to the program.

Due to the number of available participants being so low the data collected cannot be representable of the entire population of users. This evaluation should be used as a stepping stone for a more comprehensive evaluation with a large number of users from multiple schools and in multiple fields. This evaluation found GoReact is easy to navigate and use, provides adequate tech support, is efficient in reducing teacher time spent collecting and grading videos, and causes students to re-watch their videos at least twice.

 

DESCRIPTION OF PROGRAM BEING EVALUATED

GoReact is designed to be used in multiple fields of study as a tool to effectively interact with and assess video. Boise State University’s American Sign Language department employed the use of this tool, GoReact, in the Spring 2016 semester starting in January and ending in May. Initial meetings produced objectives to be met with use of the tool and were evaluated in two classes with 25 students each. Both instructors are deaf and had no prior experience with the software besides seeing it presented at an academic conference. Teachers were encouraged to use GoReact however they would like, either utilizing all features or one specifically. They were also provided links to help documents online located on the software website, email support, and video phone tech support from the organization. Students paid $19.95 each for access to the software, received no training besides what their teacher provided them, and were directed to explore and use the tool however their teacher instructed them to.

 

Program Objectives

Initial meetings with stakeholders produced the following objectives:

Usability

  • Teachers and students will find GoReact easy to navigate and use
  • Teachers and students will enjoy using GoReact finding it effective and efficient
  • Teachers and students will feel they have adequate technical support from GoReact staff
  • Teachers will use GoReact for
    • providing quality feedback on student created videos
    • allowing live presentations to receive real-time feedback
    • collecting student video responses to a stimulus video
    • managing media and learning components
    • consulting student analytics

Efficiency

  • Teachers will spend less time providing quality feedback on student created videos
  • Teachers will spend less time dealing with technology errors (USB importing, DVD quality, YouTube uploads)
  • Teachers will spend less time collecting student videos

Effectiveness

  • Students will re-watch their video with feedback at least twice
  • Students will reduce the frequency of repeated errors with the language

Impact

  • Students confidence with ASL will increase

 

Program Components

GoReact software is a cloud-based platform accessible from any device with internet access. In order for a teacher or student to use this software they need some form of video and audio capture device as well as internet access and a GoReact account.

The GoReact platform can store information and media from multiple courses as well as components for multiple assignments. Components include color-coded markers for fast and easy feedback and teacher created rubrics. Four different activities are designed into the software and available for teacher use: presenter self-record, live record, stimulus video response, and stimulus comment only. For a more in depth explanation of each activity see Appendix A. After a teacher joins GoReact they set-up their account, their courses, and their activities then invite students to join. Students pay the account fee and can access course activities as the teacher makes them available. Once a course and activities are set-up, students can access assignments and upload or record videos directly into the platform. From there, if the teacher allows, peers can provide feedback on each others videos or the instructor can access assignments and provide personal feedback via audio, video, or text comments. If a student has a question on a comment they can ask it directly within the video in a chat like sidebar. Grades are assigned and points given directly within the GoReact platform so teachers can keep track of student analytics. All videos are saved in the GoReact cloud and are accessible by students even after the course is over.

The ASL department joined GoReact and attended a short meeting to get their accounts set up and running. No Formal training was provided although embedded into the GoReact platform is a text based tour and a Help option where a teacher can ask a question in English and receive a response from a tech support representative. Teachers involved in this evaluation were encouraged, but not required, to use all or any features of GoReact as they would like or were comfortable using.

 

EVALUATION METHOD

A system analysis model was used to evaluate the program focusing on the extent to which program objectives were met. The evaluation gathered data from program participants on the usability, efficiency, effectiveness, and impact of GoReact in an American Sign Language Context. Data was collected via an anonymous online survey distributed through student and teacher email accounts. Participants were reminded to take the survey twice at the end of the semester. Each survey question was directly related to a program objective, see Appendix B.

 

Participants

Participants for this evaluation included all teachers and students in the only two classes implementing GoReact at this university. Fifty students were included, 43 females and 7 males, from two separate American Sign Language courses. They range in age from 19 to 46 years old. Teachers included are two deaf individuals, one female and one male. Another source of data was the GoReact website where organizational objectives were gathered.

Procedures

A student survey and a teacher survey were created to address all outlined objectives, See Appendix B. Both surveys were created using Google Forms and were distributed with a link via school email. Additional reminders to complete the survey were distributed twice in two week intervals. Both teachers participated in the survey and their questions and results can be seen in Appendix C. Thirty-six out of fifty students participated in the student survey and their questions and results can be seen in Appendix D. Surveys were completed during participant’s own time so there was no effect on class time or normal procedures.

 

Data Sources

The results of this evaluation are based on two surveys, a teacher survey and a student survey. The surveys addressed the usability, efficiency, effectiveness, and impact of GoReact software in an American Sign Language context. One Hundred percent of teachers and seventy-two percent of students completed the surveys.

 

RESULTS

The results of this study are based on a small number of users from one department in a mid-sized university. They do not represent the entire population of users, but merely give us an idea of the use of this software in this particular situation. Although the number of participants in this evaluation is small, these results can be used as a stepping stone to creating a more comprehensive evaluation with a larger program that better represents the entire population of users. Following are the evaluation results outlined according to factors and objectives.

 

Usability

Students and teachers were asked how strongly they agreed with the following statements addressing the usability of GoReact.

Objective: Teachers and students will find GoReact easy to navigate and use.

Both teachers agreed that GoReact was easy to navigate and use. Out of the students surveyed 63% either agreed or strongly agreed GoReact was easy to navigate while 69% either agreed or strongly agreed it was easy to use.

Objective: Teachers and students will enjoy using GoReact finding it effective and efficient.

From the usability standpoint, 100% of teachers agree that GoReact is efficient and effective and enjoy using it. When it comes to students, 70% find GoReact to be effective while 17% do not and 13% are neutral.

What is interesting here is that even though 70% of students find GoReact to be effective only 44% enjoy using it. 33% of the students are neutral on whether or not they enjoy using it and 22% do not enjoy it at all.

Only teachers were surveyed on whether or not they found GoReact to be efficient and both agreed with the statement that GoReact is efficient.

Objective: Teachers and students will feel they have adequate technical support from GoReact.

It became clear during data analysis that this question could have been asked in a two parts. The first asking if they have contacted technical support and if yes, was it adequate and timely. It is the assumption that if tech support has not been contacted, the participant would select a Neutral level of agreement. Along with this assumption, 70% of students who contacted tech support received adequate feedback while 30% did not. 

Objective: Teachers will use GoReact for providing feedback on student created videos, allowing live presentations to receive feedback, collecting student video responses to a stimulus, collecting student comment responses to a stimulus, managing components, and consulting analytics.

The results collected on this objective were gathered from a survey question with the ability to select multiple options. Each option corresponded to one of the item listed in the objective. Both teachers use GoReact for only one option: providing quality feedback on student created videos. This could be due to the short amount of time to become accustomed to the tool and its features, the lack of training with the tool, or the small amount of teacher participants.

 

Efficiency

Objective: Teachers will spend less time providing quality feedback on student created videos

It is obvious from that data that teachers are spending significantly less time providing feedback to students with GoReact then they were prior to using it. In future evaluations it should be asked what types of feedback they are providing; written, audio, and/or video. 

Objective: Teachers will spend less time dealing with technology errors.

This was an interesting find. The difference in answers between the two teachers prevents the ability to draw any conclusions in this area. Data from more teachers is required to determine if GoReact lessens the amount of technology errors experienced by teachers.

Objective: Teachers will spend less time collecting student videos

Due to the variety in responses it seems like this question was misunderstood and should have been rephrased. Teacher one said it took him/her less than an hour to collect student videos with and without GoReact while teacher two said “When I don’t use GoReact, I am less motivated to use videos.  I would assign maybe two per semester” and answered n/a for both questions regarding how long it took to collect student videos with and without GoReact. In future evaluations it should be asked in a more specific structure.

 

Effectiveness

Objective: Students will re-watch their video with feedback at least twice.

Although the objective was for students to review their videos at least twice 50% of students are re-watching their video once with 25% re-watching it twice. Without the use of GoReact students rarely re-watch their video at all. Future evaluations should include this comparison.

Objective: Students will reduce the frequency of repeated errors with the language.

This question came back inconclusive. Both teachers responded with a neutral agreement meaning that either the question needs to be rephrased, more time is needed with the software, or a different approach to gathering this data is necessary.

 

Impact

Objective: Students will increase their confidence with American Sign Language

The majority of student surveyed, 44%, are neutral to whether or not the use of GoReact helped increase their confidence. With 39% agreeing that their confidence has increased and only 17% disagreeing it can be hypothesized that GoReact has an impact on student confidence. In future evaluations another survey on the impact of GoReact should be provided at a later date with more in depth questions and analysis of hard data such as comparison of grades.

 

DISCUSSION

Boise State University, a mid-sized university in the northwest, began utilizing a contextual video feedback tool with two courses in their American Sign Language department and requested an evaluation of its use to better inform their decision on whether or not to use it with the entire department.  The usability, efficiency, effectiveness, and impact of the tool was evaluated to measure progress toward specific administrative and organizational program objectives. The evaluation showed that GoReact reached four out of the ten identified objectives. This small number may be a result of the short duration of student and teacher use of the software, lack of training with the software, and/or the low number of participants in the evaluation. Do to these limitations this evaluation should not be perceived as representative of the entire population of users but instead as a stepping stone to a more in depth and comprehensive evaluation. Following is a short discussion of the results for each factor and objective.

 

Usability

The majority of teachers and students found GoReact was easy to navigate and use, effective, and supplied adequate tech support but while both teachers enjoyed using GoReact, only 44% of students did. This is a significant find because if less than half of the students using the software enjoy using it, administration may not want to continue its use. It was also found that both teachers at this university only used one of GoReact’s features. This may be due to the short amount of time with the software, lack of training, or the small number of teacher participants (2).

 

Efficiency

The evaluation found that GoReact significantly lowered the amount of time teachers spent grading videos but came back inconclusive on whether or not it lowered time spent dealing with technology errors or collecting student videos. Reducing the time spent grading was one of the major objectives discussed at the administrative level so seeing it significantly met by the GoReact software should be of high importance when making the decision on whether or not to continue its use with the department. The other two objectives coming back inconclusive inform evaluators of the necessity for more teacher participants in future evaluations and/or the necessity to re-phrase survey questions.

 

Effectiveness

GoReact was effective in causing 50% of students to re-watch their video with teacher feedback at least twice but was inconclusive on whether or not they showed a reduction in language error. 50% of students re-watching their videos with feedback is a significant find in terms of objectives affecting the continued use of GoReact as the more times a student watches a video, the more they retain. It was assumed that students who receive written feedback separate from their video assignment rarely go back and watch their video again but future evaluations should gather data on this. Data collected on reduction in student error should be collected in future evaluations through grades and/or class observation.

Impact

The evaluation found that 44% of students were neutral to the impact of GoReact on their confidence with American Sign Language, 39% agreed that it helped increase their confidence, and 17% disagreed.  This finding may not have a strong influence on the decision to continue use with the department as more data should be collected with students who use GoReact for a longer period of time.

The evaluation found GoReact software achieved four out of the ten observed objectives: it was easy to use and navigate, provided adequate tech support, lowered grading time for teachers, and caused students to re-watch their videos at least twice. Two out of the ten objectives were unreached: students did not enjoy using GoReact and teachers only used one of GoReact’s features. The remaining four objectives came back inconclusive.

The evaluator concludes that participants need a longer duration of time with GoReact software and/or formal training to learn how to use other features. It is hypothesized that more time with the software and more knowledge of its features will increase student enjoyment.

The high number of inconclusive data is concluded by the evaluator to be a result of the low number of participants. Due to this, the evaluation is not representative of the entire population and should not be used to make any major decisions. Instead it should be used as a stepping stone to a more comprehensive evaluation. Recommendations for future evaluations include re-wording of survey questions, use of a variety of data sources, a larger amount of participants, and a longer duration of teacher and student interaction with the software.

 

 

REFERENCES

Blair A, Curtis S, Goodwin M, Shields S.(2013) The significance of assignment feedback: From consumption to construction. European Political Science. 12(2). 231-244.

 

Eksi, G. Y. (2012). Peer review versus teacher feedback in process writing: how effective?. International Journal of Applied Educational Studies, 13(1), 33.

Rodgers, C. R. (2006). Attending to student voice: The impact of descriptive feedback on learning and teaching. Curriculum Inquiry, 36(2), 209-237.

See, B. H., Gorard, S., & Siddiqui, N. (2015). Teachers’ use of research evidence in practice: a pilot study of feedback to enhance learning. Educational Research, 1-17.


 

APPENDICIES

Appendix A – GoReact Activities

Appendix B – Objectives, Factors Addressed, and Measurement Instrument

 GoReact Objective Factor Addressed Measurement Instrument
Teachers will spend less time collecting student videos Efficiency Teacher Survey: scale
Teachers will spend less time dealing with technology errors Efficiency Teacher Survey: scale
Teachers will spend less time providing quality feedback on student created videos Efficiency Teacher Survey: scale
Students will reduce the frequency of repeated errors with the language Effectiveness Existing Data: student grades
Students will re-watch their video with feedback at least twice Effectiveness Observational Analysis: scale
Students will increase their confidence with ASL Impact Student Survey: scale
Teachers and students will find GoReact easy to navigate and use Usability Student Survey: scale
Teachers and students will feel they have adequate technical support from GoReact staff Usability Student Survey: scale
Teachers and students will enjoy using GoReact finding it effective and efficient Usability Student Survey: scale
Teachers will use GoReact for providing quality feedback on student created videos, allowing live presentations to receive real-time feedback, collecting student video responses to a stimulus video, collecting student comment responses on a stimulus video, managing media and learning components, and consulting student analytics. Usability Teacher Survey: sentence completion

 


 

Appendix C – Teacher Survey Questions and Results

Survey Questions Teacher One Teacher 2
Are you hearing or deaf? Deaf Deaf
What level of ASL do you teach? ASL 202 I teach both 202 and 302.  No option to use more than one…
Please briefly explain how you collect, grade, and provide feedback on student created videos when you don’t use GoReact. Blackboard, YouTube comments When I don’t use GoReact, I am less motivated to use videos.  I would assign maybe two per semester.  I watch them then use a rubric and hand out my handwritten feedback to them after two, three weeks
Without using GoReact – How long does it take you to collect student videos?

Multiple choice in hours

Less than 1 hour n/a
Without using GoReact –  How long does it take you to provide quality feedback on student videos?

Multiple choice in hours

1-2 hours Per class and per assignment, I would guess about 4-5 hours for 20 students
Without using GoReact –  How often do you encounter technology errors when grading student videos or creating materials for class?

Multiple choice in hours

Once a project Four times a project
Using GoReact – How long does it take you to collect student videos?

Multiple choice in hours

Less than 1 hour n/a
Using GoReact –  How long does it take you to provide quality feedback on student videos?

Multiple choice in hours

Less then 1 hour Per class and per assignment, I would guess about 2 hours for 20 students
Using GoReact –  How often do you encounter technology errors when grading student videos or creating materials for class?

Multiple choice in hours

Twice a project Never

 

 

Survey Questions Teacher One Teacher 2
I use GoReact to…..

–        provide quality feedback on student created videos. (Activity One)

–        allow students, and/or myself, to give real-time feedback on live student presentations. (Activity Two)

–        collect student video responses to a stimulus video. (Activity 3)

–        collect student comment responses on a stimulus video. (Activity 4)

–        manage media and learning components. (Library)

–        consult student analytics.

provide quality feedback on student created videos. (Activity One) provide quality feedback on student created videos. (Activity One)
GoReact is easy to navigate.

Strongly Agree (1) – Strongly Disagree (5)

2 2
GoReact is easy to use.

Strongly Agree (1) – Strongly Disagree (5)

2 2
I enjoy using GoReact.

Strongly Agree (1) – Strongly Disagree (5)

2 1
GoReact is effective.

Strongly Agree (1) – Strongly Disagree (5)

2 1
Students have shown a reduce in repeated errors since I started using GoReact.

Strongly Agree (1) – Strongly Disagree (5)

3 3
GoReact is efficient.

Strongly Agree (1) – Strongly Disagree (5)

2 2
Whenever I have had difficulty with the software, GoReact has always supplied adequate and timely technical support.

Strongly Agree (1) – Strongly Disagree (5)

2 2

 

 

 

Appendix D – Student Survey Questions and Results

Question Answer Strongly Agree Agree Neutral Disagree Strongly Disagree
What is your hearing status? Hearing – 34

Deaf – 1

HoH – 1

         
What level of ASL are you in? 302 – 20

202 – 16

         
When you receive feedback from your instructor on one of your videos, how often do you review the feedback by re-watching your video? Zero – 0

Once – 18

Twice – 9

Three – 8

Four – 1

Five – 0

         
Since I started using GoReact my confidence with ASL has increased.   4 10 16 6 0
GoReact is easy to use.   11 14 5 6 0
GoReact is easy to navigate.   9 14 7 5 1
I enjoy using GoReact.   6 10 12 6 2
GoReact is effective.   10 15 5 5 1
Whenever I have had difficulty with the software, GoReact has always supplied adequate and timely technical support.   4 5 23 3 1