I. INTRODUCTION
Educators continually search for better ways to increase student learning outcomes. As the result of a study conducted by Malik and Agarwal [1] to analyse the use of multimedia in several disciplines in higher education, it was discovered that there were many pedagogical issues concerning the failure to increase students’ interest and interaction in the classroom. Hence, it was pointed out that more research was required in the field of multimedia pedagogy so that multimedia form, design and content would not hinder the normal educational process [1]. There exists limited evaluation on the use of multimedia in a blended teaching and learning environment in higher education.
The aim of this research is to evaluate student learning in different multimedia-based classrooms of blended learning in higher education using Kirkpatrick’s model [2]. The different multimedia-based classrooms include face-to-face instruction using PowerPoint slides and problem-based learning. Since the model is widely used to evaluate training in business and industry, this research aims to evaluate blended learning in higher education using Kirkpatrick’s model.
II. BLENDED LEARNING
In the literature, various definitions can be associated with blended learning. In this research, blended learning is defined as the use of different learning strategies with multimedia that include face-to-face instruction with PowerPoint slides and problem-based learning.
There is considerable support in the literature for blended learning. In a study by Hoic-Bozic [3], the blended elearning model was used which combined face-to-face environment and online learning via a Learning Management System. It was discovered that the students were in favour of the approach to learning and their academic achievements were improved compared to the previous offerings of the course that were conducted in classical manner. According to Maryam and Marlia [4], there are several advantages of blended learning over mere face-to-face instruction or online instruction through reviews of the literature, such as increased communication, engagement of face-to-face communication, sense of community, improved academic performance, etc.
III. KIRKPATRICK’S EVALUATION MODEL
In this study, Kirkpatrick’s model is used for evaluation due to its flexibility which can allow educators to determine multiple assessments and methods which also tallies with the standard practice of higher education institutions in Malaysian universities and other universities around the globe. Kirkpatrick’s model has been used extensively to evaluate training programs and as the basis for evaluation of a variety of educational settings [5, 6]. Kirkpatrick suggested four levels at which training programs should be evaluated that represent a sequence of ways to evaluate programs. The four levels are Level 1: Reaction – How did the participants perceive the program? Level 2: Learning – Did the participants manage to learn? Level 3: Behavior – Were the participants able to transfer what they have learned? Level 4: Results – What are the tangible results from the program? [5].
In order to apply this model, it is most desirable to measure from Level 1 to Level 4, but this is usually not possible. From the literature, it was found that less than 10% conducted Level 4 evaluation [7]. Therefore, for this study, Level 1, 2 and parts of Level 3 where possible, were implemented. Level 4 is not implemented in this study as it is also not relevant to the educational context that the instruction was applied [8].
IV. THE STUDY
The study conducts an evaluation using one of the principles of data collection – the use of multiple sources of evidence [9]. Triangulation is the rationale for using multiple sources of evidence. Methodological triangulation was implemented where multiple methods were used to study a single program to enhance the validity of the research findings [10]. Data were collected from various research instruments including student survey, interview and pre- and post-test. The data collection was also guided by Kirkpatrick’s model. Different data were collected according to the requirements for each level of the model.
This study sought university students where the authors were working through a convenience sample approach as the sample was obtainable or convenient to reach [11]. Participants involved students taking a course offered to students of computing, engineering and management consisting of 31 students (Malay, Chinese, Indian and International students). Demographics data were collected at the start of the course to gather data on their background, age, gender, computer self-efficacy, attitude towards the internet, previous experience using a computer technology and application, etc. The majority of the students were in the age ranges of 21 to 25 years old. The number of females in the course outweighed the number of males by one. Half of the students were Malays and in their second year of study while the majority of them were management students. The majority of the students have positive attitude towards the internet, high computer self-efficacy and have experience using a computer technology and application.
There are two types of multimedia-based learning environment - Multimedia-based classroom 1 (M1) and Multimedia-based classroom 2 (M2). In M1, the lecturer used multimedia-based content in the form of power point slides presentation which was projected from a PC, onto the white screen in front of the classroom using a projector. In M2, a project was given to them at the start of the course (in the first week of the trimester) and they had to complete it by the end of the course (14 weeks). They would form groups of four to five members and proposed a topic of their choice. Students would have to create an interactive application using multimedia technology. Throughout the development process, they had to utilize any knowledge gained from M1 as well as from other sources mainly the internet in order to transfer the knowledge to the project. Students had to create an interactive multimedia application as a final step of their multimedia project development.
A survey was given to gauge their reaction to both learning experience. Interview with 20% of the sample was conducted to obtain their perceptions. Students were also administered with pre- and post-test to assess learning that had been gained. Questionnaire was given to obtain their perception on any transfer of learning that has occurred at the end of the course as part of the evaluation guided by Kirkpatrick’s model.
V. RESULTS
Students were given a survey to gather their reaction or attitude in blended learning which is in accordance with Level 1 of Kirkpatrick’s model. The 5-point Likert scale survey (1=Strongly Disagree, 2=Disagree, 3=Not Sure, 4=Agree and 5=Strongly Agree) contained items according to the dimensions in learning that are commonly found in the literature i.e. Learning with multimedia, Motivation, Engagement, Good design and Quality of instructor and learning activity [12-17]. The survey was analysed using SPSS to determine the reliability of the survey or the Cronbach’s Alpha. The Cronbach’s Alpha coefficients were more than 0.6 in all dimensions, so it was deemed as reliable [18].
Table 1 and 2 depict the results of the survey on the student’s responses in terms of the five dimensions, with M representing the mean, p representing percentage of students’ responding positively towards the items and the standard deviation of the responses, in descending order of the mean. The grand mean and Cronbach’s Alpha are also shown for every dimension.
Results showed that students had positive reaction towards all the learning dimensions to enhance learning in M1. Students have positive reactions as the use of multimedia elements and good design of the traditional method provided them with engagement and ease of following. The quality of the instructor could help them learn the course better in face-to-face instruction using the PowerPoint slides.
It can be seen from the survey results that students reacted positively towards M2 in terms of the five dimensions to enhance learning. The findings indicate that students generally liked the environment where they could be engaged and motivated to learn with multimedia to solve a given problem. Good design and quality of learning activity like teamwork were among the important indicators of a well-received problem-based learning environment.
The experiences and perceptions of the blended learning environment as stated by a few students in interviews are reported in this section. This evaluation is in accordance with Level 1 of Kirkpatrick’s model where student’s reaction was further investigated. The interviews occurred after the end of the course. The interviews were semistructured and following the guide that had been prepared. About 20%(six) of the students were interviewed.
Students were asked to explain why they liked or disliked learning in M1. Four of them responded positively as they thought that it was the best way to learn as it was the traditional method of learning and multimedia provided better understanding. This learning could make them understand better because they could ask the lecturer to further explain or just refer to the slides and it depended on the lecturer to guide the learning process. This type of learning also served as good study notes for them so they could focus on important parts. It managed to engage them because it was conducted in a classroom where they could follow the lecturer’s pace. Nevertheless, one student didn’t really like it because it was boring sometimes and there was this one student who gave a very negative reaction by saying that she hated it. The responses can be categorised into themes of learning with multimedia, engagement and quality of instructor. Table 3 shows their responses in these themes.
Students were also asked to further explain on their likes or dislikes of learning in M2. All of them liked the environment because they could gain new experience and gain more knowledge by applying it. Their feedback can be categorised as experience/knowledge, better understanding and quality of learning activity. They could learn better with the help of other group members when working in a group. The topics that have been covered in other environments could be applied in the project. It was perceived to expose them to real world projects in the future, gained the required skills and understand all the other content that they needed to apply in order to finish the project. As such, it could test the lesson that they have acquired through the other learning environments and felt the exposure to real world industries as depicted in Table 4.
Pre-tests and post-tests were conducted for M1 while post-test only was conducted for M2. Pre-test was given in M1 because the knowledge was not totally new to the students based on the findings from the demographics survey. However, knowledge in M2 was considered new as they were never exposed to similar project before. Test is used to measure student’s learning in accordance with Level 2 of Kirkpatrick’s model. The normality test is used to determine whether the sample has a normal distribution before testing the significance of the mean score for the test. As recommended for sample sizes that are less than 50, it is more reliable to use Shapiro-Wilk test [19]. Table 5 and 6 show that the p value is below 0.05 which is statistically significant, hence it is assumed that the sample is not normally distributed.
| Shapiro-Wilk | |||
|---|---|---|---|
| Statistic | Df | Sig. | |
| Pre-test | .944 | 31 | .108 |
| Post-test | .910 | 31 | .013** |
| Shapiro-Wilk | |||
|---|---|---|---|
| Statistic | Df | Sig. | |
| Post-test | .911 | 31 | .013** |
Therefore, after the pre-tests and post-tests have been conducted for M1, their means are compared using Wilcoxon Signed-Ranks Test whereas for M2, One-sample Wilcoxon Signed-Ranks Test is used to test the significance of the means against the hypothetical mean of 10.00 (50% of post-test score). It is assumed that the means are equal to the medians when using this test. Table 7 and 8 show the results of the tests.
| Pre-test(M =8.06) | |
| Post-test | Z = −4.896a |
| (M =10.87) | p = 0.000** |
| Hypothetical Mean (M = 10.00) | |
| Post-test | Z = −2.069a |
| (M = 10.90) | p = 0.039** |
Using this method, statistical significance is accepted where p is less than 0.05 (p<0.05) when testing for 95% confidence. Hence, there is a statistically significant difference of pre-test/hypothetical mean and post-test in M1 and M2 as p values are less than 0.05. This indicates that students have actually made significant progress in their learning process after being exposed to this blended learning environment. This result is similar with existing literature where students have shown significant learning. This result is consistent with their positive reaction found in Level 1.
In order to gain insight on whether students have transferred the knowledge from the lectures to the project and learning outcome, questionnaire was used to obtain their perception on their transfer of learning according to Level 3 of Kirkpatrick’s model. Most of them felt that all or some of the learning has been transferred. There were a small number of students who did not feel that they have transferred the knowledge due to various reasons including slow learner and confusion.
Students perceived that all of the knowledge was transferred to the project and learning outcome as they could relate and apply the concepts learned in the face-toface instruction using PowerPoint slides. The content learned was applied into the problem-based learning and learning outcome was perceived to be achieved:
-
Student 1: “Of course, we have applied all of the principles on the project”
-
Student 2: “Yes because it made me understand more and apply animation from face-to-face”
-
Student 3: “Yes I’ve transferred the principles and technique learned”
Students felt that they could only transfer some of the knowledge because they wanted to keep it simple or did not understand how to apply in the project and learning outcome:
-
Student 1: “Yes but not all as we keep our work simple and use less colour”
-
Student 2: “Some sort of it. We use basic animation only”
-
Student 3: “Yes, I think half only because cannot understand some parts to apply”
However, there were five students who did not feel that they have transferred the knowledge but did not give any comment, not sure or due to being slow learner or it was confusing:
VI. CONCLUSION
The study has demonstrated that it is possible to evaluate a blended learning environment using Kirkpatrick’s model by combining several methods of data collection. Overall, students were generally satisfied with the learning environment. They were found to significantly learn from the test results. They also perceived that they were able to transfer the knowledge gained from the lectures. As such, research contribution that resulted from this study include the deeper insights into assessments in multimedia-based blended learning environment and the empirical evidence on the evaluation of the learning environment in higher education that has been demonstrated based on the three levels of Kirkpatrick’s model. Although there are some limitations in this study, it has provided empirical evidence on views, attitudes, learning and knowledge transfer of students in blended learning.
