RUNNING HEAD: Impact of Web Resources
Impact of Web-Based Integrated Course Resources
in Introductory Psychology
Charles M. Slem and Ned W. Schultz
Psychology and Human Development
Cal Poly, San Luis Obispo
cslem@calpoly.edu
Introduction
The World Wide Web has been touted as the most recent technological breakthrough in the teaching-learning enterprise. While there has been a proliferation of models of different presentation formats for computer-based course resources, there has been very little systematic study of the impact of specific models for web-based resource delivery (Shneiderman, 1997). In addition, research on computer-based instructional assistance is mixed (Forsyth & Archer, 1997), with fewer students using these course resources than anticipated.
The goal of this study was to assess the impact of one model of web-based course assistance as it was being implemented during Fall quarter, 1996, to complement a traditional large introductory psychology course. The web project created a platform which was designed to integrate the lecture component of the course with the accompanying textbook resources. Special design considerations were made for simplicity, functionality, and a student oriented perspective that linked course resources with student goals. This was consistent with a review of the literature by Atkinson and Kydd (1997) who found that computer usage depends on factors such as perceived usefulness, ease of use, and computer ability.
The lecture component of the course web resources contained lecture outlines, links to library resources, related campus resources, sample exams, instructor based resources (e.g. Heaven's Gate Cult page), and relevant World Wide Web links. The text book component included chapter outlines, concept synopses, graphs and figures, definitions of key terms, links to related concepts in other chapters, sample test questions, World Wide Web sites that pertained to specific topics, access points to web or CD based tutorials, and additional commentary. Students could seamlessly navigate from the instructor developed lecture resources to the supporting textbook resources. A particular topic covered in the lecture portion of the course was linked to the web resources developed to support the textbook's treatment of the same topic. Supporting resources, e.g. sample test questions, links to library components, etc. were also conveniently linked to the text and lecture components.
Method
Because of potential access and equity issues, use of the web resources was not mandatory. Students could meet class objectives without depending upon the web resources. Most of the web resource material was made available in paper format at the university library. Aside from a five minute demonstration and encouragement to see the instructor or a campus computer lab assistant if students had any problems, no further training was given.
The 500 students enrolled in this course were divided into smaller groups which met once a week in a recitation. In the first author's recitations, 156 students were administered three surveys during the quarter as part of their recitation assignment. Performance on exams was also collected.
The first recitation survey was administered in the first class meeting and returned at the next class meeting. Students responded to questions about demographics, computer competence, access to the web, and attitudes about instructional technology, including computer anxiety.
The second survey was administered just after the results of the first midterm were discussed in recitation. Using the following Likert type scale, students responded to questions about access to the web pages, ease of use, and perceived usefulness:
The last survey was administered in the ninth week class meeting and returned at the last class meeting of the quarter. At this point in the quarter all assignments except for the final exam had been completed. Students responded to questions about the usefulness of each of the web components, reactions to the World Wide Web, impact on their web competence, and attitudes about instructional technology as it related to the course.
Students assessed the usefulness of the web resources on the following 7 point Likert type scale:
Final Examination
In addition to the in-depth surveys given to the approximately 30% of the students, two survey questions were added to the end of the final examination in the course to obtain a general estimate of overall class usage as well as an evaluation of the usefulness of the web resources after all the course requirements were completed. All 500 students enrolled in the class had the opportunity to respond.
Results
Characteristics of Students
Of the 156 students who were assessed in greater depth, 148 completed and returned their surveys, 2 chose not to participate at all, and 6 failed to turn in the first survey but completed subsequent surveys.
The vast majority of students, 75.2%, were freshman. There were relatively few upper class-level students (10.2%). Consistent with the large number of first year students, the students were young, mean=18.7 years. The majority were male (54.7%).
Two-thirds (66.7%) of the students owned their own computer, with 59.5% considering themselves at least competent with computers. Upper class-level students tended to rate themselves as more competent than lower class-level students, r=.26 (p<.01). Men were more likely to consider themselves competent than women, t=2.58, df=146, p<.012.
A large percentage of students considered themselves as web novices, 45.6%, and only 33.5% saw themselves as at least competent in using the web. Women were less likely to see themselves as web competent as men, t=3.51, df=145 p<.001.
At the beginning of the term, 73.5% of the students reported having access to the World Wide Web. Over 60% of the students recognized that they had access via the campus computer labs. Approximately 40% had access via their own computer.
Students had very favorable attitudes concerning instructional technology. Combining "strongly agree" and "tend to agree", the majority (55.5%) believed that class assignments with computer technology would significantly increase their learning, although fully a third were undecided. A large percentage of students, 71.1%, believed that adding World Wide Web pages to supplement a class should enhance their educational experience. Over 90% of the students were hopeful that computers would provide more access to resources and faculty.
Almost half the students, 49.3%, reported that the use of technology would give them more personal control, and less than 20% believed that computers would increase their level of frustration. In both instances, however, a significant number of students were undecided. While 76.7% of the students generally believed that being trained to use new computers or new applications should be easy, 73.3% reported having some anxiety about learning how to learn to operate new programs. Computer anxiety was very modestly related to class standing, r=.18 (p<.05), with lower class-level students reporting more anxiety.
In summary, although these primarily freshmen students had positive attitudes about this technology, almost half were novices and most reported some general apprehension about learning how to operate new computer based programs.
Survey 2: Reactions After Exam I - Third Week
The second survey was administered in the class period following the first exam.
Use Patterns at the Third Week
Students indicated the frequency of their use of individual web components ("1 = a lot", "2 = a little", or "3 = not at all"). Table 1 reveals that the majority of students used most of the web resource components. Several components were used more heavily than others. Over 80% of the students made use of the sample tests and almost 80% used key term pages to some extent. Almost 65% of the students used the chapter synopses and links to definitions of terms with which they were not familiar. Just over half the students used links to the internet, and less than half (47.5%) used the few Web Wizard tutorials that were available.
Access Issues at the Third Week
The majority of students, 64.4%, found that it was relatively easy to get a computer to use the course web pages, however, almost a quarter of the students reported at least some difficulty gaining access to a computer, and 6.3% reported great difficulty in gaining access.
Exploring the course web pages had been a frustrating experience for 15.4% of the students, and 53.1% thought it would have been helpful to have a demonstration in recitation to show students how to use the web pages.
Perceived Usefulness of Resources at the Third Week
Lecture Resources
Using a 5 point Likert scale, students rated the usefulness of each of the web resources. Combining "Strongly agree" and "Tend to Agree", 64.8% of the students found the lecture outlines useful, 77.5% found the sample lecture questions useful, and 72.5% found the links from the lecture to the text resources useful.
Students who used the resources rated the usefulness of the resources more highly. For example, of the students who used the sample test questions heavily, 60% strongly agreed to the statement about the test's usefulness compared to only 25% of the students who used the sample questions either "a little" or "not at all."
Text Resources
Using the 5 point Likert scale, students also rated the usefulness of each of the text web resources. Combining "Strongly Agree" and "Tend to Agree," over 70% found the synopses, charts, definitions and sample test questions useful. As with the lecture sample examinations, heavy users rated the sample exams higher than those who used the sample exams little or not at all (70.3% to 26.9%).
Web Resources and Performance at the Third Week
There were few obvious relationships between use of the web resources and performance on the first exam. Using a composite index of use produced a correlation of -.06 (ns) with test performance. Use of the sample examinations alone and test performance produced a correlation of r=.11 (ns). Students who used the test resources heavily had a higher average (39.7) than the groups that only used the resources a little or not at all (37.6 and 38.8). However, students who did not use the sample tests at all did better than those who used it "a little".
In summary, at this early point in the quarter most students found that despite some access and training issues, the web resources were easy to use and useful. Almost 90% of the students still thought the idea of an "electronic syllabus" was worth pursuing, and over 85% could see how the course web pages could enhance their understanding of the class material.
Ninth Week Survey
At this point in the quarter all assignments except for the final examination had been completed.
Usefulness of Text Resources
Table 2 summarizes student ratings of perceived usefulness of individual web resources, using the following 9 point Likert type scale:
Most mean scores clustered around the "2" (useful) rating point. Text questions and learning objectives were most highly rated (average less than 2), and World Wide Web oriented links were lowest rated (average almost 4).
In order to exaggerate the distinctions between resources, a more restricted definition of usefulness was applied in Chart 1 by combining the ratings of "A Must" and "Useful" and eliminating "Somewhat Useful". Using the more restricted definition of usefulness, 84.3% of the students rated the sample test questions as useful, and three-fourths rated the learning objectives and text outlines as useful. Over half the students saw definition links, key terms pages, and synopses as useful. Links to charts and explanation links were rated useful by between one-third and one-half of the class. The web links and the web controversy links were seen as useful by less than 20% of the students.
Usefulness of Lecture Resources
Table 3 summarizes student ratings of perceived usefulness of individual lecture web resources. Most mean scores clustered around the "2" (useful) rating point. Lecture questions and learning objectives were most highly rated (less than 2), and cartoons and web oriented links were lowest rated (3.7 and 3.8).
As with the text resources, a more restricted definition of usefulness was applied in Chart 2 by combining the ratings of "A Must" and "Useful" and eliminating "Somewhat Useful". Even when using a more restricted definition of usefulness, the majority of lecture resources were rated as useful by students.
Within this more restricted definition of usefulness, 84.3% of the students rated lecture test questions as useful, and 73.9% rated the sample recitation paper as useful. A written paper on advertising was a recitation assignment. Lecture outlines and the course syllabus were rated useful by almost two-thirds of the class. The web links and links to campus resources were seen as useful by less than 20% of the students.
Overall Perceived Impact of Web Resources on Performance
Students assessed the impact of web resources on their performance in the course using the following five point scale:
Combining the "strongly agree" and "tend to agree" categories, almost 70% of the students believed the web pages increased their learning in the course (Table 4). Approximately 9% of the students "disagreed" or "strongly disagreed" that the web resources increased their learning. Almost 70% (69.7%) believed that the use of the web resources gave them more control over their work.
The belief that the web resources increased learning was correlated with "My learning style is well suited for computers," r=.38 (p.<001). While there was a slightly negative relationship between initial web competence and the perception that the web pages increased learning, r=.11 (ns), students who reported the greatest gains in web competence were more likely to believe the web pages were helpful in their learning, r=.23 (p<.01).
Using this class as a demonstration, 83.9% of the students believed that adding web resources to supplement classes would enhance their educational experience. This belief was related to their perception that the use of the introductory psychology web resources increased their learning of the course, r=.38, (p<.001).
Effect of the Web Resources on Web Competence
Students responded to a question about the effect of using the introductory psychology web resources on their current level of web competence ("much more competent", "somewhat more competent", "competence remained the same"). Table 5 reveals that 46.5% of all students, and 53.2% of the novices, reported that they were much more competent as a result of using the web resources.
In addition to using the web resources to fulfill the requirements of evaluating them, 57.8% saw the web resources as a tool to give them access to a wide variety of course resources, 66.9% considered the web resources as a means to help them master the course content in order to get a good grade in the class, and 29.6% reported that the tool would help expand their awareness of a variety of psychology topics through the World Wide Web.
Effect of Course Web Resources on Computer Anxiety
In the initial survey of student characteristics, almost three-fourths of the students reported anxiety over having to learn new computer programs. Table 6 summarizes the change at the ninth week. Using the a five point scale described above (with "1" representing "strongly agree") in response to the statement "I am anxious about having to learn how to operate new computer programs," the average score changed from 2.19 to 2.52. These results represents a significant reduction in computer anxiety (t=-3.06, df=136, p=.001). Inspection of Table 6 indicates that the number of students who strongly agreed with the statement dropped from 29% to 16%. Of those students who reported that they were web novices at week 1 and filled out the final survey, n= 63, the mean changed from 2.10 to 2.49, (df=62, t=2.72, p=.004).
Survey Questions on the Final
Two survey questions were added to the end of the final examination in order to assess the entire class after all the assignments in the course were completed. All students taking the final exam had the opportunity to respond to these questions. Of the 478 test booklets returned, 404 (84.5%) contained responses to the evaluation questions. Students returned 74 blank test booklets. Analyses were done on the 404 responses.
Table 7 summarizes the reported frequency of use. Of the 404 students who filled
out the questionnaire, 24.6% reported that they used the web resources "a lot,"
85% reported using the resources at least "a little," and 15% reported
not using the web resources at all.
Table 8 summarizes the student evaluation of the usefulness of the web resources.
Over 60% rated the web resources as very useful, and over 94% of the students found
the web resources were at least somewhat useful. Only 5.6% of the students rated
the web resources as not useful at all.
Frequency of use was related to usefulness ratings, r=.56, (p<.001). For example, of the students who reported using the web resources "alot," 92.9% reported that the web resources were "very useful".
Discussion
The results of this study indicate that the initial version of the introductory psychology web resources was heavily used and considered a very useful tool in the mastery of introductory psychology. Using this class as a demonstration, 83.9% of the students believed that web resources to supplement classes would enhance their educational experience. It is clear that the application of web resources to integrate the lecture and text portions of the class is highly desirable, and this model of delivery of web-based resources for introductory psychology courses was effective.
The initial characteristics of the students in the class indicated that while these students had a positive attitude about technology to support the introductory course, there a surprising number of novices and a large number who were anxious about having to learn new computer applications. In this light, it may be surprising that such a high percentage of students actually used the resources by the end of the quarter. However, as Anderson and Hornby (1996) found, student attitudes towards computers does not necessarily affect the potential benefits of computers in psychology classes. Fortuitous design considerations and functionality from the students’ perspective appear to be likely reasons for the high usage in the current study.
The web pages were designed to be as straight forward and self-instructing as possible, and served to provide web novices with a helpful introduction to the use of the web-based resources. Complex eye catching graphics were avoided as were interactive tutorials which ran the risk of "crashing" student computers. That is not to suggest that resources were dry and sterile. One of the hallmarks of the World Wide Web at the time was its inherent playfulness and creativity which were incorporated into the design of these web resources but did not distract the student from successfully navigating the various components.
The high ratings of ease of use suggest that if well designed, web resources should be a very easy to learn computer application. Over 90% of novices reported gaining more competence in using web resources by the ninth week. Even more important is the indication that concern about learning new computer applications went down significantly over the course of the quarter. Although this was not an experiment (which would have controlled for any additional computer experiences that students may have had in other classes), the use of the web resources for this introductory course is a likely contributing factor to the students’ improved attitudes about learning new computer applications.
One might anticipate that level of web competence by entering students should have improved since 1996 when 45.6% of the students were novices. That has been the case. By Fall of 1997, the number students enrolled in this class rating themselves as novices had dropped to 16.5%, and in a similar class during Summer, 1998, the percentage of novices fell to approximately 7%. However, the lean graphics and straightforward design of the course web resources are still appreciated by the students.
The second likely factor for the success of the web resources was their functionality. Individual components were designed and integrated with attention to typical student tasks as a primary guide for development. The organization of the web allowed the student to clearly see how the resource or activity was directly related to their specific course’s content and how that specific resource would enhance their mastery of the content for examination purposes. This was especially true for the textbook resources which paralleled the structure and content of textbook. This level of integration makes this particular model unique. The typical publisher based web resources have lacked the direct correspondence to their texts’ concepts.
Students believed that the web resources significantly increased their personal control of the course resources as well as their learning of introductory psychology concepts. Not only does the 24 hour availability of a wide variety of resources increase one’s sense of control, the web resources were integrated in such a way that students could use those aspects of the web resources that they personally felt were most productive to them.
In general, students rated more highly those resources which were more directly associated with achieving higher grades in the course (e.g. examination performance). This partially explains why World Wide Web links, cartoons, and links to campus resources were not seen as useful by the majority of students. These resources were not necessary tools in the mastery of the concepts for examination purposes.
The current quarter system did not lend itself to self-directed enrichment activities, although 30% considered the resources a way to expand their awareness of a variety of topics in psychology. In a class that has very few psychology majors, this suggests that even though students did not take full advantage of that feature they were aware of it. When access and equity issues are fully resolved, web-based assignments may be more practical and links to the World Wide Web may be seen as more central to their success in the course.
The mixed relationship between use of web resources and performance is problematic but not atypical. Forsyth and Archer (1997) found little evidence of general academic gain in a technology enhanced course, but found some indications that academically weak students might be beneficiaries of technology enhancement. To pursue this possibility with the web resources presented in this paper, a subsequent pilot study using the current introductory psychology web resources (but with a relatively small n) found that SAT scores were related to test scores, students with lower SAT scores more heavily used the web resources, and that those lower scoring SAT students who used the web resources did better than those lower scoring SAT students who did not use the web resources. Further investigation with larger numbers of students is currently underway. It is possible that web resources may be an especially useful tool for students who struggle with traditional course tools.
Another possible reason for the failure to find an effect of the web resources was the hard copy availability of those resources. Because of equity and access issues, there were paper copies of most of the web resources which students had access to in the campus library. Additional publisher provided study guides were also available to students. Students had access to a wide array of tools to enhance their learning, and this was not an experimental study with rigorous controls.
The success of this model of web resource delivery has significant implications for publishers of introductory psychology texts. Publisher web sites have typically included resources that are not tightly tied to the tasks of students who are using the publisher’s textbook as part of their introductory psychology experience. One of the prime reasons for the success of the model described in this study is that the text resources paralleled the text concepts closely enough so that the student could see the context for a particular activity or component. Until web resources are clearly perceived by students (and faculty) as being directly relevant to their goals for the course, the promise of the web will remained unfulfilled.
References
Anderson, M., and Hornby, P. (1996). Computer attitudes and use of computers in psychology courses. Behavior Research Methods, Instruments, & Computers. 28, 341-346.
Atkinson, M, and Kydd, C. (1997). Individual characteristics associated with world wide web use: An empirical study of playfulness and motivation. The Data Base for Advances in Information Systems. 28, 53-62.
Forsyth, D. and Archer, A. (1997). Technologically assisted instruction and student mastery, motivation, and matriculation. Teaching of Psychology. 24, 207-212.
Shneiderman, B. (1997). Designing information-abundant web sites: issues and recommendations. International Journal of Human-Computer Studies, 47, 5-29. Retrieved August 8, 1998 from the World Wide Web: http://www.hbuk.co.uk/ap/ijhcs/webusability/shneiderman/shneiderman.html
| Resource | Usage | |||
| "Alot " | "A Little" | "Not at All" | ||
|
Sample Exams |
45.5% |
38.5% |
16.1% |
|
|
Key Terms |
38.5% |
40.6 |
21.0 |
|
|
Synopses |
17.5% |
46.2 |
36.4 |
|
|
Charts |
17.0% |
43.3 |
39.7 |
|
|
Web Links |
8.5% |
43.0 |
48.6 |
|
|
Tutorials |
5.7% |
41.8 |
52.5 |
|
Mean |
Must+Useful (1+2) |
Sum Useful (1-3) |
Uncertain (4) |
Sum Not Useful (5-7) |
|
Text Questions |
1.8 | 84.3 | 92.9 | 5.0 | 2.1 |
Text Outline |
2.1 | 75.0 | 92.1 | 17.1 | 4.5 |
Synopses |
2.4 | 55.4 | 87.8 | 9.4 | 2.8 |
Definition Links |
2.4 | 65.0 | 82.9 | 12.9 | 4.3 |
Key Terms Page |
2.2 | 64.7 | 92.1 | 3.6 | 4.3 |
Chart Links |
3.0 | 34.5 | 75.5 | 12.9 | 11.6 |
Explanation Links |
2.7 | 45.0 | 76.4 | 17.1 | 6.4 |
Web Links |
3.8 | 17.1 | 46.4 | 25.0 | 28.6 |
Web Controversy |
3.8 | 13.7 | 46.0 | 31.7 | 22.3 |
Learning Objectives |
1.9 | 74.3 | 95.0 | 2.9 | 2.1 |
Mean |
Must+ Useful (1+2) |
Sum Useful (1-3) |
Uncertain (4) |
Sum Not Useful (5-7) |
|
Lecture Outline |
2.3 | 64.3 | 95.7 | 17.1 | 4.5 |
Link to Text |
3.1 | 33.1 | 71.2 | 17.3 | 11.5 |
Lecture Questions |
1.8 | 84.3 | 92.9 | 12.9 | 2.1 |
Links to Web |
3.7 | 18.7 | 46.8 | 3.6 | 23.8 |
Link Campus Resources |
3.6 | 18.7 | 49.6 | 2.9 | 24.5 |
Email to Prof |
3.2 | 35.5 | 60.3 | 27.7 | 12.1 |
Syllabus on Web |
2.4 | 67.8 | 83.9 | 7.7 | 8.4 |
Sample Paper |
2.1 | 73.9 | 91.6 | 5.6 | 2.8 |
Learning Objective |
1.7 | 83.9 | 97.2 | 1.4 | 1.4 |
Cartoons |
3.8 | 26.6 | 44.8 | 22.4 | 19.6 |
Mean |
Strongly Agree ( 1) |
Tend to Agree (2) |
Agree (1+2) |
Hard to Decide (3) |
Tend to Disagree (4) |
Strongly Disagree (5) |
|
|
Increase Control
|
2.2 | 24.6% | 45.1% | 69.7% | 21.1% | 8.5% | .7% |
|
202 Web resources enhance learning
|
2.1 | 25.9 | 47.6 | 73.4 | 17.5 | 7.7 | 1.4 |
|
Web pages will enhance other classes
|
1.9 | 37.1 | 46.9 | 83.9 | 9.1 | 6.3 | .7 |
|
Web replace text
|
3.4 | 5.6 | 14.0 | 19.6 | 29.4 | 39.9 | 11.2 |
|
My learning style suited to computers
|
2.8 | 6.4 | 37.6 | 43.9 | 34.0 | 14.2 | 7.8 |
|
Training should be easy
|
2.3 | 19.6 | 49.7 | 69.2 | 20.3 | 7.0 | 3.5 |
|
Computer anxiety
|
2.5 | 16.1 | 41.3 | 57.3 | 22.4 | 14.0 | 6.3 |
|
Won’t replace teachers and classrooms
|
2.4 | 27.3 | 28.7 | 55.9 | 26.6 | 16.1 | 1.4 |
|
Computers will increase frustration
|
3.4 | 7.0 | 12.0 | 19.0 | 27.5 | 41.6 | 12.0 |
Much More Competent |
Somewhat More Competent |
Competence Remained the Same |
|
All Students |
46.5% | 33.1% | 20.4% |
Novices |
53.2 | 38.7 | 8.1 |
Mean |
Strongly Agree ( 1) |
Tend to Agree (2) |
Agree (1+2) |
Hard to Decide (3) |
Tend to Disagree (4) |
Strongly Disagree (5) |
|
|
Initial Anxiety
|
2.19 | 28.8% | 44.5% | 73.3% | 9.6% | 13.0% | 4.1% |
|
Final Anxiety
|
2.52 | 16.1 | 41.3 | 57.3 | 22.4 | 14.0 | 6.3 |
Number |
Percent |
|
"A lot" |
99 |
24.6% |
"Some" |
160 |
39.8 |
"A Little" |
83 |
20.6 |
"Not at All" |
60 |
14.9 |
Number |
Percent |
|
"Very Useful" |
230 |
61.8 % |
"Somewhat Useful" |
121 |
32.5 |
"Not Useful" |
21 |
05.6 |