Professional Documents
Culture Documents
David Jones Faculty of Informatics and Communication Central Queensland University d.jones@cqu.edu.au Sandy Behrens Faculty of Informatics and Communication Central Queensland University s.behrens@cqu.edu.au
Abstract
Due to the potential benefits there has been widespread interest in the use of online assignment submission and management (OASM). However, much of the reported work is limited to small-scale use usually in courses taught by innovative staff. This paper draws on experience with OASM in a growing number of courses taught by a number of different staff from different disciplines. It aims to investigate the process by which OASM is adopted across an organization and identify the factors that influence its growth, adoption, and adaptation. In particular the paper demonstrates that the introduction and successful use of OASM requires changes in practice, the ability to cater for a wide range of requirements, and a continual questioning of the appropriateness of the resulting changes to practice.
1. Introduction
Assessment for many University courses includes some form of assignment which students complete, individually or as groups, and submit for marking and feedback. Assignment management involves the collection, date stamping, redistribution to tutors for marking, collation of results, and return of assignments to students [1]. Tregobov [2] breaks the process down into four stages: submission, recording, marking, and return. Online assignment submission and management (OASM) involves the use of the World-Wide Web, the Internet and computers in this process. There are a number of problems with traditional approaches to the submission and management of student assessment, especially in a distance education setting [3]. Application of OASM promises to address many of the problems of traditional approaches and offers the potential to adopt a wide range of new practices. Consequently, there has been widespread interest in OASM [1-20, 2830]. However, much of that literature is of a limited nature. Usually describing the features, characteristics and initial use of these systems in a limited number of courses taught by a small number of innovative staff that are usually the authors of the subsequent papers.
Implementing technology, even implementing technology well, provides no guarantee that the system will be used or be effective [23]. There is a long history of failed technology-based innovations in education [24]. Many such projects fail due to the innovators underestimating the consequences of new technologies [26] and failing to accommodate environmental and contextual factors affecting implementation [22]. This paper seeks to contribute to the understanding of OASM by proposing a model that encapsulates the issues, challenges and opportunities encountered when OASM is adopted by a wider range of faculty. This model has grown out of the use of OASM within the Faculty of Informatics and Communication (Infocom) at Central Queensland University (CQU). An important aspect of the model is that the adoption and use of OASM is an ongoing, evolutionary tale that is continually driven by these issues, challenges and opportunities. As a result any decision, which limits that evolution, can limit future benefits and adaptation. The paper commences with a summary of the potential benefits of OASM and previous work in the area. It then introduces the model that seeks to explain the factors driving the evolution of OASM. The following section uses the model to describe Infocom's experience with OASM since 1994. Finally, the paper briefly discusses the implications of this model.
2. Previous work
All teaching and learning requires administrative support [21]. Much of this administrative support is to some degree transparent but if performed inefficiently can become immediately obvious and can distract students and staff from the learning process [1]. Online Assignment Submission and Management (OASM) is one administrative task that can consume much of a course coordinator's time [1]. Reducing the amount of time consumed, addressing other problems, and benefiting from new practices are some of the reasons that have driven many staff to adopt and use OASM. Darbyshire [1] reports finding over 6000 hits from web search engines for pages describing online assignment submission procedures to students. In
addition there is a large amount of literature written on the topic. This section outlines some of the reasons driving the adoption of OASM and provides an overview of much of this previous work.
greater the automation used the greater the reduction in marking time. For a number of reasons achieving accuracy in assessment is vital [19]. Any problems, perceived or otherwise, with the accuracy of assessment reduces student confidence. A CQU student reports: I had 2 assignments marked incorrectly and another one the results were not recorded at all (this went unnoticed till I saw my final result), it makes me wonder just how many other errors are slipping by unnoticed. Appropriate reporting mechanisms combined with online assignment management increase the transparency of the marking process by allowing students to view the progress of their assignment. Systems with built-in automatic logging of the process increase accountability [17]. Automation of manual processes such as transcription of results can help reduce errors. Electronic marking of assignments also removes problems associated with the illegibility of handwritten marker comments [17]. A CQU staff member reports on a related problem: a tutor in Melbourne went missing, this was a paper-based submission, and just disappeared with about ten assignments and I had no marks and I had no assignments either. If they had been electronically submitted it would have been really easy just to pull it off the web page and send it down to the other marker... Another staff member adds: Just the flexibility, you know, if a marker makes a real hash of things we can jump in and do it ourselves or get someone else to do it...... Upon graduation most CQU graduates from the regional campuses move away to capital cities. This means that there is a limited population of qualified graduates in regional areas. This can make it very difficult to find appropriate markers for advanced level courses. Use of OASM offers a solution to this problem as one Infocom staff member discovered. another advantage is that one of my students is now a graduate. He's now working in Canberra. He's keen to help me out from that distance, so he's going to do some marking for me. Large class sizes make it necessary to introduce a large number of markers. Maintaining consistency between multiple markers is a major problem in large classes [20]. Mason and Woit [15] believe that online marking of assignments can lead to improved marking consistency and integrates well with online mark reporting. Moderation of assignment marking is an increasing problem at CQU where an individual course can have over 10 markers spread geographically throughout Australia and South-East Asia. OASM can help in the moderation process as two Infocom staff found.
...an electronic copy of their assessment is really, really handy because I can do parallel moderation with the marker, so the marker can be away marking and I can be moderating or assessing what I think the marks would be, and I can actually look at the difference, so there's no lag to the moderation. OASM enables a single submission process that can be used by all students regardless of physical location. As one Infocom staff member found It creates an equality between on-campus students and flex students no matter where they are because they all want to be able to submit at the same time. So theres nothing like as long as your assignment is post-marked by Monday you will be accepted in when the assignment was due in by Friday afternoon. There is an equality in that, it will be Friday afternoon, your assignments are here instantaneously, no excuse otherwise. And I think that they detect that as well, they all know that theyve all got the same deadline. So thats good. Paper-based submission introduces a number of other problems as two Infocom staff report also just not dealing with stacks of paper, you stick it on a disc and take it home, that's your marking on a disc and you can take it home rather than this great pile of rubbish. Yes, well there's still assignments upstairs that the student's haven't collected. In fact, there are still a heap left over in that box there from Winter term, people never bothered to come back and collect them. At least when you email them all back, they're gone. A study by the Center of Academic Integrity at Duke University in 1999 found that 68% of the 2,100 students polled said that they had committed at least one academic offense such as plagiarizing [25]. CQU academics, like many around the world, believe the ease of access to information provided by the World-Wide Web along with a number of other factors is increasing the levels of student plagiarism. Definitely, definitely there is a general feeling that we have problems, plagiarism problems, on the International campuses where we are identifying a number of incidents.......So, I'm sure we're all looking for methods for dealing with that problem. However, using the information processing capabilities of computers, drawing on electronic copies of student assignments submitted via OASM, and using access to Web search engines detecting plagiarism can be a relatively straightforward task. There is a growing range of available systems, both commercial and free, that enables staff to perform plagiarism detection [27].
Apart from detecting plagiarism OASM also offers the potential to avoid it. With support from the technology and appropriate assignment design it is possible to generated individual specialized assignment items that allow students to collaborate without fear of plagiarism [14].
3. The model
Drawing on existing publications [3, 4, 5], results of student surveys, interviews with staff, and system statistics a range of factors have been identified as influencing the evolution of OASM within Infocom. Those factors have been categorized and the model shown in Figure 1 is an attempt to represent their relationship.
society in which the organization operates. Legal requirements, perceptions of technology, and other issues from the wider society can also inhibit or encourage OASM evolution.
Society. During 1995 the major Australian telecommunications company introduced a special long distance rate for phone calls late at night that made Internet access considerably cheaper for students. Infrastructure. The course coordinator had a personal UNIX server already hosting email lists for the department that was used to host mail folders. Organizational. Individual academics had the freedom to innovate and experiment within their courses that allowed the adoption of email based assignment submission. New System. As a result of these factors the use of email for the submission of assignments was made optional for distance students in this single course. Assignment files were attached to messages and sent directly to the course coordinators email address. The assignments were manually forwarded to and from markers and students. Only 20 of the 54 distance students made use of email assignment submission.
submission process was also used in the 1996 offering of that initial course. Technology. During this time GUI-based Internet tools were becoming available and interest in the Web was becoming more prevalent. The combination of increasing numbers and increasing Internet access led to over 50 students using the system during the second half of 1995 and the first half of 1996. Individuals. Due to the relationship between the two courses the majority of students using this system in 1996 had used the system previously. The course coordinator now had increased experience with online assignment submission. In particular, the importance of changing practice and the value of sufficient training was known. A new marker with greater familiarity and access to technology was employed. Infrastructure, Organizational, and Society. Remained much the same during this period.
versions of the Internet Explorer Web browser caused problems since it did not support the file upload standard. As a result students using Internet Explorer had to revert to manual email submission. Course. OASM was still being used in the same two courses. Both courses now had large accompanying web sites and OASM was made compulsory for both courses. Individuals. Staff and students are becoming more familiar with the Web and related technology. The marker now had two or three semesters experience with online marking. By 1997 all students had access to the Internet from home or work. Students are starting to expect 24/7 availability of Websites. In the second half of 1996 the coordinator had time relief to develop web-based tools and systems to support other staff. Organizational. The importance of Internet technologies to the operations of the University was starting to be realized. Along with time relief for the originator of OASM a departmental web server is purchased. However, no long term funding is provided for maintenance that made maintaining a 24/7 operation required significant unpaid effort. A partnership with a commercial company leads to the addition of new campuses and growing student numbers. The academic structure of the organization is restructured. Society. Many small Internet Service Providers (ISPs) are becoming available. Infrastructure. A new faculty server is available however uptime is a problem. There is growing problems with the number of University provided dial-up modems available for students. Students slowly start moving towards using ISPs.
This integration enabled the system to know which courses a student was enrolled in and also to make use of a global username/password available to all students. Courses. During this time staff teaching other courses started becoming interested in using OASM. In the first half of 2000 over 770 students in 6 different courses used OASM. Individuals. Other academic staff became involved for the first time. These new staff and their markers did not have experience with OASM. During this time the creator of the OASM system spent time on a number of other projects so during this four year period little work was done on the OASM system beyond its initial integration with the CQU student records system. The marker involved in the 1996-1997 OASM system joined CQU's central Information Technology division working on CQU's student record system. Organizational. Primarily due to the past connections outlined in the previous paragraph a good relationship existed between the academic staff and CQU's central Information Technology division. This relationship made it much easier to negotiate access to their databases. Gaining permission from the data owner was more problematic and after many formal approaches was achieved by interrupting the head of that division at lunch. Infocom employed a Webmaster and expended more money on a larger Web server. All Infocom courses now had a primitive, generic website. Numbers at international campuses grew until they eventually provide more than 50% of all students in courses. Consequently there are more staff, both teaching and markers, involved with courses. Society. Larger, nationwide companies were beginning to replace smaller, local Internet Service Providers (ISPs). Local call access to the Internet becomes available to all parts of Australia. The Internet and World-Wide Web became widely popular. Australian federal and state governments were under pressure to better support regional Australia. Technology. By 2001 a student without Internet access in computing courses becomes the exception. The Web has arrived has a major tool for students and the ease-of-use of the available tools had dramatically increased. Infrastructure. During this time University infrastructure did not keep up with the growth in popularity. Problems were caused by too few dialup modems for student use, limited bandwidth, too few support staff, and under-resourced servers.
appropriate increase in supporting infrastructure. This caused considerable difficulties and contributed to a negative experience for many students. The new staff members limited experience with OASM combined with the infrastructure problems meant that average turnaround times for OASM assignments were usually somewhat more than the average turnaround time for the traditional method. New System. Significant money was spent by CQU to improve infrastructure including both bandwidth and servers. The major changes to OASM were the addition of extra services such as copy detection, reporting, and support for multiple markers. Organizational. A support team of four people has been set up within Infocom to support the use of OASM and other online technologies. This is a move initiated by the Faculty Dean and this alone enables further development of the system. One head of school asks for a presentation about OASM to be made to her staff. This encourages more staff to adopt OASM. This increase is mainly due to the growing problems with the multicampus operations of CQU and the administrative headaches it causes for teaching staff. CQU adopts a new student records system that requires significant changes to the OASM system. The requirement that staff at international campuses mark their student's assignments is written into contracts and thus must be supported by OASM. The new server equipment is shared amongst three different parts of CQU and requires some work for this relationship to operate appropriately. Courses. By 2002 OASM is being used in 18 different courses offerings ranging from 1st year courses in computer programming and web development, through 2nd year Multimedia Design and Human Computer Interface courses, to post-graduate courses in database design. Individuals. Greater than 50% of the students in most Infocom courses have English has a second or third language. The majority of the courses using OASM are first year courses. These factors combined require a continual effort to improve the interface and operation of OASM. Staff concerns about plagiarism grow with students found to be adopting practices such as selling past assignment solutions on E-Bay. Each staff member that adopts OASM brings with them their own vision of how assignment management, marking and return should be performed. Infrastructure. The large University server is more than sufficient for current requirements. Bandwidth to some of the campuses is limited and cause problems. Technology. Access to copy detection software is available during 2002. The growing use of OLE objects and images in student assignments lead to students submitting very large RTF files. In mid-2002 virus detection software is made available on the main server
for the first time. This enables the replacement of RTF files with Word documents. Society. The Australian government strengthens reporting requirements for full fee paying overseas students (those studying at CQU's international campuses) to include a requirement that the University must be able to identify students who have not submitted assessment. This requires the addition of a reporting mechanism for administrative staff. The government also embarks on a push to increase quality assurance systems within Universities. Due to the governmental auditing requirements there this has sparked an increased interest in OASM due to the automated tracking of assignments at all stages.
5. Discussion
A wide range of factors, as described by Figure 1, has influenced the development of OASM within Infocom. The ability for Infocom's OASM system to evolve in response to these factors has resulted in a system that better serves the needs of students, staff, and the organization. Such evolutionary development has only been possible through the presence of appropriately skilled individuals supported through sufficient resources, infrastructure, organizational policies, and a flexible technology.
This raises questions about the suitability of integrated course management systems such as WebCT and Blackboard. Appropriately resourced in-house products are able to provide more functionality than these integrated systems [1] and be modified in response to unique local conditions. Since no company will adequately address all the possibilities for online assessment it is critical that such a system have open system architectures that support interoperable modules [12]. Course management system vendors are recognizing this fact by adopting more open architectures. The need for evolutionary development of OASM is not limited to the technology and its features. Experience with OASM has shown that initially, faculty attempt to recreate existing practice with the new technology. This is often because changing both practice and medium is seen as being too difficult. However, as experience grows staff begin to question basic assumptions and start to look at using the unique characteristics of OASM and alternate assessment approaches. If OASM systems accommodate only classical approaches they will constrain instructors and not allow the development of more innovative approaches [12]. A system that only accommodates "innovative" approaches will not allow staff to adapt gradually [12].
[3]
[7]
[8]
[9] [10]
6. Conclusions
Drawing on experience with online assignment submission and management (OASM) since 1994 this paper has developed a model that encapsulates seven factors that have contributed to the evolutionary development of OASM. It has used that model to show how OASM within an individual organization has changed from submission via email in a single course by 20 students to an integrated, web management facility incorporating support for copy detection and distributed markers used in a single calendar year in 18 different course offerings by 3500 different students.
[11]
[12] [13]
[14] [15]
7. References
List and number all bibliographical references in 9point Times, single-spaced, at the end of your paper. When referenced in the text, enclose the citation number in square brackets, for example [1]. Where appropriate, include the name(s) of editors of referenced books.
[1] P. Darbyshire, Distributed Web-Based Assignment Management, Web-Based Learning and Teaching Technologies: Opportunities and Challenges, A. Aggarwal, Ed. Hershy, PA: Idea Group Publishing, 2000. A. Tregobov, The Web-Based Assignment Submission Systems, presented at NAWeb'98, University of New Brunswick, Canada, 1998.
[16]
[17]
[2]
[18]
D. Jones, B. Jamieson, Three Generations of Online Assignment Management, Proceedings of ASCILITE'97, R. Kevill, R. Oliver, R. Phillips, Eds. Perth, Australia: pp 317-323, 1997. D. Jones, Solving some problems of University Education: A Case Study, Proceedings of AusWeb96, R. Debreceny, A. Ellis, Ed. Gold Coast, pp 243-252, 1996. D. Jones, Solving some problems with University Education: Part II, Proceedings of Ausweb'99, A. Ellis, Ed. Balina, Australia, 1999. R. Byrnes, B. Lo, J. Dimbleby, Flexible Assignment Submission in Distance Learning, presented at the IFIP World Conference on Computers in Education VI, WCCE'95, 1995. M. Chi-sang, Y. Kin Sun, C. Siu Leung, L. Chow, The Electronic Learning Environment for IT-based course in Hong Kong, Proceedings of EdMedia'99, R. Oliver, B. Collis Eds., AACE, Seattle, USA, June, 1999, p13841385. G. Burnett, K. Burnett, Integrated Online Course Delivery System: Florida State University School of Information Studies, Proceedings of EdMedia'99, R. Oliver, B. Collis Eds., AACE, Seattle, USA, June, 1999, pp 1511-1512. R. Laws, Totally Integrated Internet Courses, Proceedings of EdMedia'99, R. Oliver, B. Collis Eds., AACE, Seattle, USA, June, 1999, pp 787-791 S. Young, M. McSporran, R. Dewstow (2000), Suffering Remotely: Challenges when teaching on-line, Proceedings of EdMedia'2000, J. Bourdeau, R. Heller Eds., pp 1590-1591 J. Palmer, R. Williams, H. Dreher, Automated Essay Grading Applied to a First Year University Subject - How can we do it better?, Proceedings of Informing Science, InSITE - "Where Parallels Intersect", June 2002 M. Hopper, Assessment in WWW-Based Learning Systems: Opportunities and Challenges, Journal of Universal Computer Science, 4(4), 1998, pp 330-348 P. Davies, S. Hansen, G. Salter, K. Simpson, Online assessment with large classes: issues, methodologies and case studies, Proceedings of WebNet99, AACE, Honolulu, USA, 1999, pp 1498-1499. J. English, P. Siviter, Experience with an Automatically Assessed Course, Proceedings of ITiCSE'2000, ACM, Helsinki, Finland, 2000, pp 168-171. D. Mason, D. Woit, Providing mark-up and feedback to students with online marking, Proceedings of the 30th SIGCSE Technical Symposium on Computer Science Education, ACM, 2000, pp: 3-6. B. Oliver, G. Mitchell, Setting the PASE - The Value of Computer Aided Assessment, Proceedings of the First Australasian Conference on Computer Science Education, ACM, Sydney, Australia, 1996, pp 103-110. B. Price, M. Petre, Teaching Programming through Paperless Assignments: an empirical evaluation of instructor feedback, SIGCSE Bulletin, 29(3), September 1997, pp 94-99. P. Summons, J. Coldwell, C. Bruff, F. Henskens, Automating Assessment and Marking of Spreadsheet Concepts, Proceedings of ACSE'97, Melbourne, Australia, 1997, pp 178-184.
[19] J. Preston, R. Shackleford, Improving On-line Assessment: an Investigation of Existing Marking Methodologies, Proceedings of ITiCSE'99, Cracow, Poland, 1999, pp 29-32 [20] J. Preston, Evaluation software: improving consistency and reliability of performance rating, Proceedings of ITiCSE'97, ACM, Uppsala, Sweeden, 1997. [21] D. Jones, C. McCormack, Class Management: The Forgotten Task, Proceedings of the Third International North American Web Conference, University of New Brunswick, Canada, 1997, pp 109-125. [22] D. Jonassen, Designing Constructivist Learning Environments. Instructional Theories and Models. C. M. Reigeluth. Mahwah, NJ, Lawrence Erlbaum, 1998. [23] R. Kling, J. Allen, Can Computer Science Solve Organizational Problems? The Case for Organizational Informatics, in Computerization and Controversy, Rob Kling (editor),, Academic Press, San Diego, pp 261-276 [24] T. Reeves, A Research Agenda for Interactive Learning in the New Millennium. Proceedings of EdMedia'99, R. Oliver, B. Collis Eds., AACE, Seattle, USA, June, 1999. [25] D. Quan, Universities combat rising Internet plagiarism, The Ottawa Citizen, 16 Feb. 2001, final edition. [26] L. Sproull, S. Kiesler, Connections: new ways of working in the networked organization. Cambridge, MIT Press, 1991. [27] J. Young, The Cat-and-Mouse Game of Plagiarism Detection, The Chronicle of Higher Education, July 6, 2001, A26. [28] M. Roantree, T. Keyes, Automated collection of coursework using the Web, Proceedings of ITiCSE'98, Dublin, Ireland, Pages 206-208 [29] M. Joy and M. Luck, Effective electronic marking for on-line assessment, Proceedings of ITiCSE'98, Dublin, Ireland, Pages 134-138 [30] D. Jackson, A Semi-Automated Approach to Online Assessment, Proceedings of ITiCSE'2000, Helsinki Finland, 2000, pp 164-167.