Instructional Design Project Evaluation

This in depth evaluation demonstrates my ability to assess the adequacy of student learning and evaluate the effective implementation of educational technologies.

Part 4. Evaluation

4a. Evaluation Plan

 In order for a project to be successful evaluation must be woven throughout. An effective evaluation plan will provide the designers and instructors with the information necessary to adjust and improve the project. The evaluation plan for this project is described here and uses Kirkpatrick’s Four-Level Evaluation Model found in Larson and Lockee (2014).

 

Level I: Learner Reaction

During the seminar, the instructor will be available to the learners and take note of any questions they ask about the material. The instructor should also take note of how the learners are reacting to the content in order to improve instructional materials for future seminars. After the seminar is completed, a survey will be provided in order to measure their reaction to the material covered in the seminar. Suggested questions include:

  1. Did the instructional materials (videos) help you in learning the content better then if you learned it on your own?
  2. In your opinion was the ASL used in the videos clear and easy to follow?
  3. Did the videos cover the content adequately? If not, what do you feel could use more attention?
  4. Did you use the Learner Objective checklist? Was it helpful to your learning process?
  5. Were you able to get your questions answered by the seminar instructor?
  6. Were you able to use this newly learned content in your classroom with ease? If not, what would assist you in doing this?

 

Level II: Learning

The learning that takes place in this seminar is best assessed using performance tasks due to it’s pass or fail nature.  And also due to this pass or fail nature, there are no numerical comparisons of pre and post seminar learning. Therefore, the learning that took place during this seminar will be evaluated using a survey method that requires participants to use the skills they learned in the seminar. The survey should be sent via a Google Doc where participants will transfer it to their personal drive and answer the questions using the comment feature.  Suggested questions for this survey include:

  1. Did you successfully transfer this document to your personal Google Drive?
  2. What are the ways you can share this document with others?
  3. Have you used Google Drive in your classroom?
  4. How is your Google Drive organized? How are your folders labeled?
  5. If I receive a link to a Google project from a student, where can I find it in Google Drive?

 

Level III: Behavior

 To evaluate how the skills learned in the seminar are being transferred to the participants day-to-day tasks, a survey will be provided. The suggested questions given here are to be sent to the participants of the seminar, but another idea is to survey the students of the seminar participants and ask about the teacher’s confidence level with Google Drive. Some suggested questions for the survey sent to the participants of the seminar include:

How often do you use Google Drive after the seminar compared to before?

  1. Do you feel confident sending and receiving shared documents?
  2. During your day-to-day tasks, how often do you turn to Google Drive for work with documents, sheets, and presentations?

 

Level IV: Results

 Come time for the next American Sign Language Teacher Association’s Fall seminar, the overall results of the previous seminar can be adequately evaluated. During this seminar tasks involving Google Drive’s share and collaborate features can be utilized. Judging weather or not the seminar was a worthwhile option can be aided by the following questions:

Did the teachers successfully transfer a shared project to their personal Google Drive?

  1. Did the teachers successfully transfer a personal project into a shared folder?
  2. Do the teachers collaborate via comments and chat?
  3. Are the teachers Google Drives organized?
  4. Do the teachers seem confident with Google Drive?
  5. Are the teachers asking a lot of questions related to Google Drive navigation and collaboration?

 

 

4b. Expert Review

 There are two Subject Matter Experts working on this project. The first is a Deaf woman by the name of Davina Snow, the head of the American Sign Language department at Boise State University. She was to review the use of American Sign Language in the learning materials and received the project on April 29, 2015 with returned comments on April 30, 2015. The second Subject Matter Expert reviewing the project is Reggie Walters, a PhD candidate in the Geoscience department of Boise State University. He is an avid Google Drive user and recommended throughout the department when it comes to technology tools and questions. Mr. Walters received the project and survey on April 3rd, 2015 and returned comments on April 20, 2015.

 

4c. Evaluation Survey

The survey found here: http://goo.gl/forms/CD40NGbzOA includes the following survey questions sent to Mr. Walters to assist in his evaluation of the project:

  1. How can the goal of this project better meet the needs of novice Google Drive users?
  2. Which objectives adequately guide instruction toward the goal and what are some other objectives that could better assist this process?
  3. What steps, if any, are missing in the instructional videos?
  4. What skills, if any, are missing from the learner objective checklist that you feel novice Google Drive learners should know?
  5. What, if anything, is unclear in the instructor written guide and how can it be adjusted to improve the quality of the seminar?
  6. Please describe your overall impression of the project and any changes you feel should be made.

 

4d. Expert Review Results

 The survey results returned by Mr. Walters were mainly positive with a few suggestions related to content. In his opinion, the goal of the project is clear and adequately touches on the needs of novice learners’ as do the objectives. When it came to the instructional videos, both Mr. Walters and Mrs. Snow had the same recommendation: a more professional background and appearance. Mr. Walters proceeded to recommend adding some common mistakes into the videos to show the learners what to avoid doing. Besides the videos, Mr. Walters was overall impressed with the project as the instructor guide, videos, objectives, and overall goal relate nicely and are easy to follow hitting all commonly asked questions related to Google Drive.

 

4e. Comments on Change

Two major changes were suggested and are in the process of being addressed. The first is the appearance of the videos. The American Sign Language use is clear but the background and appearance will be addressed. The second change that will be added to the project is the inclusion of common mistakes; accidentally deleting a shared document, what happens when you move an item out of a shared folder, and how to counteract these mistakes.

Higher Education Success Factors: A Literature Review

I completed this literature review for my supervisor. I applied research methodologies to not only enhance my own practice but to help solve issues with students dropping out of online programs.

Higher Education Success Factors – Word Document

A Literature Review: Online Higher Education / Online Teacher Education Success Factors

Databases used in this review were Google Scholar and Boise State University’s Albertsons Library EBSCOhost. Search terms included: Online Teacher Education Program Success Factors, Online Education Success Factors, Online Higher Education, Online Teacher Education, Teacher Education Program Online, Online Graduate Program Success, Graduate Student Success Factors in Online Teacher Education Programs, Graduate Student Success in Online Programs, Online Teacher Professional Development Success Factors, Online Teacher Education Case Study, Online Teacher Education Success and Failure Factors and Identifying Online At-Risk Graduate Students. Articles were excluded if they were not available in fulltext, they did not contain new information, or if they did not contain specific factors contributing to success or failure in online higher education or online teacher education. The search was limited by the time frame of 2008-present and all related articles on the first five pages of the search were included.

Factors identified as either predicting success or failure can be categorized into the following levels: Administrative, Instructor, Course, and Student.

At the administrative level, factors that contribute to the success of an online program consist of; supplying adequate technological support for teachers (Baran & Correia, 2014; Cain Phillip, Ting, Gonzalez, Johnson, & Galy, 2013;  Crawford-Ferre & Wiest, 2012; Harrell, 2008), up-to-date technological devices, and proper professional development opportunities (Cain et al., 2013), while providing pedagogical support (Baran & Correia, 2014) and access to a relevant community of practice (Cain et al., 2013; Baran & Correia, 2014; Baran, Correia, & Thompson, 2011). Reducing the teaching load on instructors or allowing use of a teacher’s aid are other ways to improve success rates of an online program at the administrative level (Cain et al., 2013).

At the level of instructor, factors contributing to the success of an online program consist of providing students with abundant involvement including: quality interactions, support, feedback, and evaluations (Hart, 2012; Cain et al., 2013; Crawford-Ferre & Wiest, 2012; Lee & Choi, 2011; Macfadyen & Dawson, 2010), supplying a well structured and designed course (Lee & Choi, 2011; Cain et al., 2013; Baran & Correia, 2014; Hart, 2012; Crawford-Ferre & Wiest, 2012), and developing an online pedagogy different from that of a traditional face-to-face classroom (Baran et al., 2011).

At the course level, factors recommended for a successful online program include an online orientation for students (Harrell, 2008; Crawford-Ferre & Wiest, 2012), a frequently asked question page, and a resource page. Course assignments should include specific expectations and instructions, as well as adequate context and abundant audio/visual aids (Crawford-Ferre & Wiest, 2012). Multiple forms of content exploration both synchronous and asynchronous should be available for various student learning styles (Crawford-Ferre & Wiest, 2012; Harrell, 2008; Terrell, Snyder, & Dringus, 2009; Perry, Boman, Care, Edwards, & Park, 2008). Course communications are recommended to be plentiful and varied utilizing both synchronous and asynchronous methods (Crawford-Ferre & Wiest, 2012; Cain et al., 2013; Falloon, 2011, Terrell et al., 2009), and provide formal and informal options for student interaction (Crawford-Ferre & Wiest, 2012; Terrell et al., 2009).

At the student level, factors found to contribute to success in an online program are time management skills (Hart, 2012; Cain et al., 2013; Dray, Lowenthal, Miszkiewicz, Ruiz‐Primo, & Marczynski, 2011), self efficacy, a positive locus of control (Hart, 2012; Dray et al., 2011; Lee & Choi, 2011; Harrell, 2008), and support from peers (Hart, 2012; Lee & Choi, 2011; Cain et al., 2013; Terrell et al., 2009). Other student factors that were found to impede on success are learning style (Hart, 2012; Harrell, 2008; Perry et al., 2008), lack of basic computer skills (Hart, 2012; Lee & Choi, 2011; Dray et al., 2011; Harrell, 2008), limited access to resources or technology (Hart, 2012; Dray et al., 2011), a lower overall GPA (Lee & Choi, 2011; Harrell, 2008), and less academic and/or professional experience (Lee & Choi, 2011). The critical student factor for failure in an online program was a feeling of isolation (Hart, 2012; Cain et al., 2013; Crawford-Ferre & Wiest, 2012, Falloon, 2011; Terrell et al., 2008). Multiple forms of communication and instructor presence are recommended for students feeling isolated including adequate amounts of student to student interactions as well as student to instructor and student to content (Crawford-Ferre & Wiest, 2012; Falloon, 2011; Terrell et al., 2008; Harrell, 2008).

A review of administrative, instructor, course and student factors contributing to the success of online programs can assist in predicting students at-risk for failure enrolled in online programs. Multiple studies attempt to identify these students using data from pre-enrollment surveys gauging student readiness (Dray et al., 2011; Harrell, 2008), to a connectedness scale developed to determine the amount of isolation a student is feeling (Terrell et al., 2008). The most effective method reviewed is an analysis of LMS tracking data following the amount of student to student interactions via discussion boards, student to instructor interactions via a mail component, and student to content interactions via self assessment attempts. The latter of the three predicting with 81 percent accuracy students that received a failing grade although tracking data must be customized to reflect the pedagogical intent of the course under study (Macfadyen & Dawson , 2010).

 

An, Heejung, Sangkyung Kim, and Bosung Kim. “Teacher perspectives on online collaborative learning: Factors perceived as facilitating and impeding successful online group work.” Contemporary issues in technology and Teacher Education 8.1 (2008): 65-83.

A study analyzing data collected from 24 students of a virtual graduate school of education program on factors that affect online group work. Data was collected via an open ended online survey and analyzed using a quantified qualitative method.  Factors identified as facilitating successful group work include: individual accountability, affective team support, the presence of a positive group leader, consensus building skills, and clear instructions. Factors identified as impeding successful group work include: lack of individual accountability, challenges inherent to virtual communication relying solely on written language, technology problems, unclear instructional guidelines, different time zones, lack of a positive leader, and lack of consensus building skills. While individual accountability was considered to be a critical factor in both successful and unsuccessful group work, the perceived importance of affective team support leaves the authors asking, “ Is affective team support a more critical factor when the course is held in an online environment?”.

 

Baran, E., & Correia, A. (2014). A professional development framework for online teaching. TechTrends, 58(5), 95-101.

A professional development framework for online teaching, developed from previous research and literature, emphasising the support from teacher, community, and organizational levels necessary for successful online teaching in higher education. An in depth analysis of how to support online teachers at each level is given: at the teaching level emphasis is put on technological, pedagogical and design support, at the community level communities of practice and peer support are emphasised, and at the organization level a strong organizational culture is necessary for a successful online program.  Along with support from each level, factors identified as contributing to the success of online courses include: time invested on planning and organization, efforts put into managing courses, increased teaching presence, and increased social presence.

 

Baran, E., Correia, A., & Thompson, A. (2011). Transforming online teaching practice: critical analysis of the literature on the roles and competencies of online teachers. Distance Education, 32(3), 421-439.

A literature review, analysis, and synthesis on the topic of online teachers’ roles and competencies revealing that for an online program to be successful, not only do the roles of teachers need to adapt to online environments but also teacher pedagogies. The authors put emphasis on the reason for failure of online courses being the replication of traditional roles and competencies from face-to-face to online environments and “onesize-fits-all

preparation and support programs for online teachers”. The review ends with a suggestion for teacher preparation and development programs to encourage a collaborative culture of online teachers practicing critical reflection, pedological inquiry, and problem solving in order to create their own online teacher personas, and in turn, successful online courses.

 

Cain, M., Phillip, S., Ting, S. R., Gonzalez, L. M., Johnson, J., Galy, E., et al. (2013). An Exploration of Students’ Experiences of Learning in an Online Primary Teacher Education Program. Journal of Online Learning & Teaching, 9(3).

A study examining the experience of eight students in an online undergraduate education program offered through The University of the West Indies. Although they had to take three prerequisite courses online to familiarize themselves with the format, all eight students, as well as the supporting faculty, were new to online courses. The study revealed that while students choose to take online classes due to convenience and flexibility, there were other factors that contributed to their success such as personal attributes: commitment, planning, and time management, peer support: collaboration, peer tutoring, and emotional support, support form other sources: Open Campus site personnel, and site technicians, online tools: forums, YouTube videos, WebQuests, chats, and wikis, teleconferences: real-time interaction and feedback, and course materials: course design and layout. Factors found to impeded on the success of the students were lack of time management, inadequate instructions, feelings of isolation, lack of feedback, too many course activities, lack of face-to-face instruction, and technological challenges. Overall it was found that for an online course to be successful students must be self-driven and organized and the course must include adequate amounts of collaboration and instruction.

 

Crawford-Ferre, H. G., & Wiest, L. R. (2012). Effective online instruction in higher education. The Quarterly Review of Distance Education, 13(1), 11-14.

A literature review and summary of effective practices in online pedagogy revolving around three categories: new methods of course design, interaction among course participants, and instructor preparation and support. In the category of course design, the review suggests the following for implementation of successful online instruction: compatible technology supporting international formats, technology support for teachers and students, an online orientation of the LMS for students, a frequently asked questions and helpful resources page, multiple methods of content exploration (both synchronous and asynchronous), and multiple options for communication including formal (structured) and informal (unstructured) settings. Following are suggestions in the category of interaction among course participants: abundant collaboration/communication opportunities between students (both asynchronous and synchronous), substantial instructor involvement, adequate content and context for assignments, specific information about expectations, and greater use of audio/visual aids.

In the last category, instructor preparation and support, the authors suggest: proper professional development, training, or preparation specific to online teaching and technology, access to proper technology and technical support, reduced teaching load or the use of a teacher’s aid, and access to a community of practice for additional support.

“More research is needed on how to prepare and support online instructors. Research should also be conducted on student experiences, motivators for participation, and perceptions of relative strengths and weaknesses of various aspects of online education”

 

Dray, B. J., Lowenthal, P. R., Miszkiewicz, M. J., Ruiz‐Primo, M. A., & Marczynski, K. (2011). Developing an instrument to assess student readiness for online learning: A validation study. Distance Education, 32(1), 29-47.

An article reviewing the development of a survey that assess whether or not a student will do well in an online class based on personal characteristics and technology capabilities. The survey was created using three phases: develop and review, item analysis, statistical analysis of reliability and validity. The personal characteristics that were indicative of a successful online student are individual beliefs in their ability to complete a college degree, beliefs about responsibility in problem solving (academic and technical), self-efficacy in writing and expression, orientation to time and time management, and behavior regulation for goal attainment. The technology capabilities portion of the survey measures basic technology skills such as the ability to use email and the Internet, as well as material access to technology, such as devices and bandwidth, and the nature and frequency of technology use. The survey seems valid and reliable though the article does not list specific characteristics or technology capabilities that predict a successful online student.

 

Falloon, G. (2011). Making the connection: Moore’s theory of transactional distance and its relevance to the use of a virtual classroom in postgraduate online teacher education. Journal of Research on Technology in Education, 43(3), 187-209.

A study analyzing the effect of a synchronous virtual classroom on student satisfaction. The study uses the lens of Moore’s theory of transactional distance when observing student satisfaction. Moore’s theory has three factors which are dialogue: any form of interaction that is of high quality and is efficient in solving student problems, structure: the level of the course’s rigidity or flexibility including clear instructions and objectives, and learner autonomy: the students’ sense of independence in the course. The third factor is contingent on the first two as a lack of structure or dialogue can increase a students dependence on the instructor. The study announces that a critical factor of student failure in online courses is their feeling of isolation due to lack of the three factors. The goal of the study was to observe the effect of a synchronous virtual classroom on students experience with this feeling of isolation. The study revealed that dialogue improved but structure declined due to students lack of flexibility, therefore learner autonomy decreased as the students were more dependent on the virtual class meeting times. In conclusion, a successful online course will offer optional synchronous virtual classroom meetings for students needed extra support in the dialogue category.

 

 

Harrell, I. L. (2008). Increasing the Success of Online Students. Inquiry, 13(1), 36-44.

An article outlining three broad categories that increase success of online students; student readiness, student orientation, and student support. To improve retention rates of online programs it is recommended that institutions employ a readiness survey to gage student attributes as they relate to success in online courses such as learning style, locus of control, computer skills, and self-efficacy. Following a student’s GPA, the existence of an online orientation was the second greatest factor in predicting online success. Technical and personal around the clock support is another factor in predicting student success.

 

Hart, C. (2012). Factors associated with student persistence in an online program of study: a review of the literature. Journal of Interactive Online Learning, 11(1), 19-42.

A literature review focused on factors contributing to student persistence and/or success in an online program. Factors identified include: flexibility, asynchronous format, time management, goal commitment, quality of interactions and feedback, satisfaction and relevance, self-efficacy and personal growth, social connectedness or presence, and support from peers. Factors identified as impeding on success are: learning style, basic computer skills, difficulty in accessing resources, isolation and decreased engagement, lack of computer accessibility, non-academic issues, and poor communication. This is an in depth literature review close to our topic of interest.

 

Lee, Y., & Choi, J. (2011). A review of online course dropout research: implications for practice and future research. Educational Technology Research and Development, 59(5), 593-618.

A literature review and analysis identifying 44 factors that contribute to student dropout of online courses. These 44 factors were organized into three categories: student factors, course/program factors, and environmental factors each with subcategories.  The first category, student factors, has four subcategories, academic background: lower GPA equates to higher dropout rate, relevant experience: more academic or professional experience equates to lower dropout rate, relevant skills: higher management skills meant lower dropout rate and lower computer skills ment higher dropout rate, and psychological attributes: positive attitudes toward school, the course, instructor, other students, locus of control, motivation and self efficacy meant lower dropout rates. The second category, course/program factors, has three subcategories, course design: well structured and relevant equated to lower dropout rates, institutional supports: lack of administrative structure, evaluation, and student support equated to higher dropout rates, and interactions: higher teacher-student interactions meant lower dropout rates as with higher student-content, but surprisingly student-student had no effect.  The last category has two subcategories, work commitments: more work commitments increased dropout rate,  and supportive study environments: higher amounts of emotional support from family and friends resulted in lower dropout rates as with a comfortable place to study and financial aid. This is a great study for our topic as it addresses factors contributing to failure unlike most other studies which focus on success factors.

 

Macfadyen, L. P., & Dawson, S. (2010). Mining LMS data to develop an “early warning system” for educators: A proof of concept. Computers & Education, 54(2), 588-599.

An analysis of LMS tracking data to identify at risk students according to their engagement with the online course. The study is looking a student-student, student-faculty, and student-content interactions identifying three predicting factors; number of forum postings, mail messages sent, and assessments completed. The model accurately predicted 81% of students to receive a failing grade. measures of total time spent online correlate only weakly with student final grade. visualizations of student tracking data for a selected course must be highly customizable to reflect pedagogical intent.

 

Perry, B., Boman, J., Care, W. D., Edwards, M., & Park, C. (2008). Why Do Students Withdraw from Online Graduate Nursing and Health Studies Education?. Journal of Educators Online, 5(1), n1.

A qualitative study reporting students’ self proclaimed reasons for withdrawing from an online nursing program. Reasons are categorized into two sections personal reasons: life circumstances and work commitments, and program reasons: learning style and evolving career aspirations.

 

Terrell, S. R., Snyder, M. M., & Dringus, L. P. (2009). The development, validation, and application of the Doctoral Student Connectedness Scale. The Internet and Higher Education, 12(2), 112-116.

An application study of the Doctoral Student Connectedness Scale, a scale developed to identify at risk doctoral students on the grounds of connectedness. It was found higher feelings of student-student and student-faculty connectedness resulted in less attrition.

 

 

 

 

 

Evaluation Report: Meeting with Stakeholders

An initial meeting was held with stakeholders involved in the implementation of a new software at our school. We collaborated with peers and subject matter experts to analyze the software and design an evaluation plan to see if this software was meeting our desired outcomes. Here is a summary of that meeting.

Meeting With Stakeholders – Word Document

 

Baughman

Meeting with stakeholders

I am evaluating the efficiency and effectiveness of a program chosen to reduce the amount of grading time teachers spend on providing effective feedback to students on student created language videos.

Leg A = Leg A is a needs analysis designed to identify where stakeholders are currently at in regards to the program. For this leg we had a group discussion with teachers and stakeholders to identify how much time they (teachers) were spending on grading student created videos and providing feedback to students. We addressed how they currently do it and why. We also discussed the effectiveness of this feedback and how teachers desired for their feedback to be more effective in reducing the amount of student repeat in error.

Leg B = Leg B is the planned “bridge” on how to get from Leg A to Leg C. In order to reduce grading time and provide more effective feedback for students (or reduce student repeat in error) stakeholders decided to purchase GoReact, a contextual video feedback tool. Other “bridges” were discussed including a training seminar, the use of free software, and in class assignments instead of student created videos.

Leg C = Leg C is the future goals of the stakeholders. Before beginning the program the evaluator sat down and discussed with stakeholders what their goals and objectives were for the coming semester. The following goals were outlined:

  • Reduced grading time for teachers (efficiency)
  • More effective feedback for students / decrease in repeat student error (effectiveness)

Addressing figure 3.1 Program Cycle in terms of the program I plan to evaluate, The two goals have been outlined by program stakeholders: reduced grading time for teachers (efficiency) and more effective feedback for students / decrease in repeat student error (effectiveness). These were identified by doing a needs analysis of the students and teachers involved in the program. The needs analysis included a group discussion of the current issues ASL teachers a facing. During the program planning phase, the evaluator was present in order to better plan an evaluation of the avenue chosen to address the outlined goals. The chosen avenue was the purchase and use of the contextual video feedback tool GoReact. For the purpose of formative evaluation, the evaluator will have access to data analytics through the software as well as the ability to observe teachers and student using the software. A summative evaluation will be given at the end of the cycle (one semester) in the form of a survey developed by the evaluator.

 

Copyright Scavenger Hunt

This activity was created to encourage a learning environment that promotes respect for copyright and fair use when it comes to signing songs. Singing songs is a popular activity for ASL students so relating copyright to this concept is a great way to teach students about best practices when creating multimedia available to the public.

Copyright Scavenger Hunt – Link to external website