2 pages

Work in progress-A web-based system for the delivery and analysis of course concept inventories

of 2
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Work in progress-A web-based system for the delivery and analysis of course concept inventories
  Session F2H 1-4244-1084-3/07/$25.00 ©2007 IEEE October 10 – 13, 2007, Milwaukee, WI37 th ASEE/IEEE Frontiers in Education ConferenceF2H-24   Work in Progress - A Web-Based System for theDelivery and Analysis of Course Concept Inventories Gerard Rowe and Chris SmaillDepartment of Electrical and Computer Engineering, University of AucklandAuckland, New Zealand and  Abstract  - A commonly-reported observation is that someseemingly academically well-prepared students strugglewith their studies once they reach the mid-points of theirdegrees. Furthermore, these same students often reportexcessive study times for courses and appear undulystressed. We hypothesise that these students havemisunderstood core concepts. We describe work inprogress to develop a tool which delivers a test of coreconcepts in a subject and automatically marks individualstudents’ attempts. The development specifications for thistool also require the ability to analyze the results toprovide the lecturer with a quantitative measure of thelevel of class understanding over a range of core concepts.By delivering and analyzing pre- and post-tests, such a toolcan also facilitate the quantitative assessment of theeffectiveness of particular teaching interventions orstudent engagement strategies. It is also intended that thetool could be used over successive years to reliablyquantify entry standards into various courses.  Index Terms - Concept inventories, automated marking,student misconceptions, electrical engineering education. BACKGROUND Study of concept mastery is well developed in PhysicsEducation Research. Such research arguably began withHalloun and Hestenes “Mechanics Diagnostic Test”, later further developed into the well-known “Force ConceptInventory” [1]. Only recently have concept mastery studiesreceived attention in Engineering Education Research, withinventories currently being developed for several fields,including electromagnetic waves, signals and systems,strength of materials, thermodynamics, materials science,statistics, heat transfer, fluid mechanics, electromagnetics,chemistry, biology and circuits [2]. Most Concept Inventories(CI) are delivered and marked manually, often significantlydelaying feedback from large classes. Automated delivery,marking and analysis would benefit instructors significantly.Such automation is the primary goal of this project. METHODOLOGY At the University of Auckland the BE degree is a 4 year  programme. Our pilot study is centred on a Year 3 elective inTransmission Lines and Fields. Four steps are involved. Thefirst step was to develop a trial Transmission Lines and FieldsConcept Inventory (TLFCI) to fit the engineeringelectromagnetics curriculum structure at the University of Auckland. The second step was to implement this trial TLFCIon OASIS (a Web-based assessment system developed at theUniversity of Auckland) using a Question Editor (alsodeveloped “in-house”). The third step is to develop the OASIScode required for the statistical analysis of the trial TLFCIresults. Finally, a trial of the automatically-delivered TLFCI is planned for Semester 1 of 2008. DEVELOPMENT   OF   TRIAL   CONCEPT   INVENTORY The method of CI development typically described in theliterature involves a four-step process:1.   Identifying the important and difficult concepts (oftenusing a multi-round Delphi process, involving subjectexperts from a number of institutions).2.   Identifying common misconceptions for thosechallenging concepts (often using student interviews,“think alouds”, unmoderated group discussions).3.   Designing questions with one or more right answersand several distractors based on thesemisconceptions. (The design of distractors whichalign with student learning models and representcommonly held misconceptions is perhaps the keystep in concept inventory design.)4.   Validating the concept inventory through peer review, qualitative analysis, and psychometricanalysis of trial runs.Ultimately a TLFCI will be formally developed using theabove process. However, as the focus of this project was oninvestigating the efficacy of a tool for the delivery, markingand analysis of concept inventories, access to a trial conceptinventory was needed. Funding provided via a TeachingImprovement Grant enabled the authors to develop sufficientcandidate concept inventory questions for a 30 minute test.This development was carried out using input from localsubject experts and a perusal of existing electromagneticwaves and electromagnetics concept inventories.  Session F2H 1-4244-1084-3/07/$25.00 ©2007 IEEE October 10 – 13, 2007, Milwaukee, WI37 th ASEE/IEEE Frontiers in Education ConferenceF2H-25   OASIS The Department of Electrical and Computer Engineering atthe University of Auckland saw computer-based assessment asthe best way to maintain educational standards in the face of increasing workloads. Partly for reasons of cost, and partly because of perceived deficiencies in some commercial packages, the Department produced its own software package,OASIS, (which stands for Online Assessment System withIntegrated Study). OASIS is a Web-based tool used for skills practice and summative assessment. OASIS is written in thePython programming language and uses the PostgreSQLdatabase for data storage. It runs on the Linux Operatingsystem. The tool delivers individualized tasks, marks studentresponses, supplies prompt feedback, and logs student activity.OASIS comprises a large question database and server-side program that delivers questions to students, marks their responses, provides instant feedback, and records students’activities. Because the Web server carries out all processing,students need only a computer with Internet access and astandard browser, making OASIS well suited to student-centred and large-class learning. The present version of theOASIS software package has been successfully used since2003, with a prototype version being used prior to that [3]. Wenow seek to extend the capability of OASIS, beyond thecurrent areas of skills practice and assessment, further into therealm of educational research. This paper represents the firstformal reporting of this extension to OASIS. PROGRESS   TO   DATE The first phase of the project is complete. A 15-questionTransmission Lines and Fields concept inventory (TLFCI) has been developed. Both multiple choice questions (MCQ) andmultiple true/false (MTF) questions are included. We areaware that some researchers consider an MCQ-only approachto be better for pinpointing misconceptions. However, in thelonger term, we wish to be able to probe cognitivedevelopment levels as well as simply identify the extent towhich common misconceptions are held. Knowledge of multiple correct answers has been tied to the levels of learningas presented by Bloom’s Taxonomy of Educational Objectives[4]. For this reason we have deliberately included a number of MTF questions in the trial TLFCI.The second phase of the project is also complete. Theimplementation of the trial TLFCI on OASIS wasstraightforward and well within the current capabilities of OASIS. The only difference was that the TLFCI uses MCQand MTF questions (with many graphical items) whereas most practice questions on OASIS are numerical. Theimplementation of these MCQ and MTF questions didhowever necessitate a small number of changes in the OASISQuestion Editor software. If automated concept inventorydelivery tools are to be widely adopted, it is important that theentry of questions is easy and requires no special expertise.The Question Editor, supplemented with a standard software package for the production of graphics, spares instructors theneed to become familiar with the HTML mark-up language.The third phase of the project is on-going. We are currentlyworking on the development of software to analyze student performance on the trial course concept questions. The projectsoftware specifications require statistical analysis of correctanswers and also, more importantly, of misconceptions held by students. Preliminary results of this third phase will beavailable by the time of the 2007 FIE Conference.Results of the fourth phase, the Semester 1 2008 live trial, will be presented at subsequent FIE and ASEE conferences.Should OASIS be judged an appropriate tool for the delivery,marking and analysis of concept inventories, a subsequent(multi-year) project will be undertaken which will replace thetrial TLFCI with an appropriately designed and validatedTransmission Lines and Fields Concept Inventory. DELIVERABLES The primary deliverable of our research is an on-line toolwhich can automatically deliver Concept Inventories tostudents and analyze their responses. Rather than rely onstudent surveys or examination results, an instructor canobtain a quantitative assessment of the effectiveness of  particular teaching interventions and student engagementstrategies by using such a tool to deliver and analyze pre- and post-course tests of student understanding of core concepts. ACKNOWLEDGMENT This project was supported by a University of AucklandTeaching Improvement Grant. R  EFERENCES   [1]   Hestenes, D., Wells, M. & Swackhamer, G. “Force Concept Inventory”, The Physics Teacher  , Vol 33, 1992, pp 141-158.[2]   Alstrum, V.L. et al “Concept Inventories in Computer Science for theTopic Discrete Mathematics”,  Proc. ITiCSE  , Bologna, Italy, June 2006, pp 132-145.[3]   Smaill, C. “The Implementation and Evaluation of OASIS, a Web-Based Learning and Assessment Tool in Electrical Engineering”, Proc.36 th ASEE/IEEE Frontiers in Education Conference, San Diego, October 2006, pp T1F-1 to T1F-6.[4]   Rhoads, T.R. and Roedel, R.J. “The Wave Concept Inventory – ACognitive Instrument Based on Bloom’s Taxonomy, Proc. 29 th  ASEE/IEEE Frontiers in Education Conference, San Juan, November 1999, pp 13C1-14 to 13C1-18.
Related Documents
View more...
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks

We need your sign to support Project to invent "SMART AND CONTROLLABLE REFLECTIVE BALLOONS" to cover the Sun and Save Our Earth.

More details...

Sign Now!

We are very appreciated for your Prompt Action!