Gloria Rogers
Program Chair
Associate Executive Director
Professional Services
ABET, Inc.
(410) 347-7700
Judy Houghtalen
Symposium Coordinator
(812) 872-6241
Kathy Gregory
Administrative Assistant
(812) 877-8816
Fax Number
(812) 877-8931
|
|
|
April 13, Friday 9:10 - 10:00 a.m. |
Course Outcome Testing System
Alireza Farahani, National University
Olin Hall O157
Session Description
A database-driven multiple-choice testing system has been developed to automate the administration, grading and reporting of assessment tests. The system includes features such as instructor survey and student self evaluation of the course outcomes. The testing system captures both student and instructor comments and reports a tabulated summary of the results. This system is particularly useful in educational institutions with distributed campuses.
Key Words
Assessment Tools, Use of Technology to Support Assessment
Assessing the Reliability of Scoring Rubrics
Judy Shoemaker, University of California, Irvine
Olin Hall O159
Session Description
Despite the widespread use of scoring rubrics to assess program outcomes in engineering education, very few studies have explored the reliability of scoring rubrics. In this session, we will review the basics of scoring rubrics, including types of scoring rubrics, advantages of using scoring rubrics, and how to develop them. Then we will review different methods of assessing inter-rater reliability of scoring rubrics. Examples from scoring rubrics used to assess senior design projects in engineering will be discussed, and recommendations for improving inter-rater reliability will be shared.
Key Words
Assessment Tools, Quantitative Methods
Outcome-Based Grading System
Francis Aldrine Uy, Mapua Institute of Technology
Olin Hall O167
Session Description
A grading system is a process by which a teacher arrives at the symbol or number that is used to represent a student’s achievement in a subject area. An outcome-based grading system is a process on which students are assessed based on how they satisfy set of course outcomes. It uses word modifiers to describe the degree of a student achievement or performance and then it is compared to the required or expected outcomes enumerated in the course syllabus. It requires a set of outcomes with qualitative words like mastery, proficient, knowledgeable, familiarity, etc. It also require a standard rating scale to determine what percentage shall we qualify student performance. The next step is to create software that will help promote the principles of an Outcome-Based Grading System. The software will be named Outcome-Based Assessment System Software (O-BASS). Upon entering all needed inputs, it will automatically create the needed reports for assessment and presentation.
Key Words
Assessment Tools, Measuring Objectives, Measuring Outcomes, Qualitative Methods
Oral Assessment for Improvement and Change
Vidar Gynnild, Norwegian University of Science and Technology
Olin Hall O169
Session Description
This paper reports on a case study of learning and academic achievement in engineering education. The purpose of this study was to learn about students’ learning through two sets of oral exams along with ensuing individual semi structured interviews. Most often exams are used for grading purposes, but oral examinations are also particularly well suited to probe students’ understanding and thereby offer unique opportunities to learn about learning. The small number of students (8) in the optional module in Oceanography enabled an in-depth study of individual strategies and approaches to learning.
Key Words
Assessment Tools, Faculty Involvement, Measuring Outcomes, Results for Improvements
Area Level Assessment Using WebCT
Milica Markovic, California State University Sacramento
Olin Hall O257
Session Description
The integration of current assessment practices at CSUS that include industrial board, class level assessment, feedback from graduating seniors, alumni surveys, alumni site visits and topical assessment process will be presented. Specifically an inexpensive, easy to apply, fast and secure assessment tool implemented in WebCT will be presented. The test does not require any additional work from faculty. Analysis of test results over several semesters will be discussed and compared to some of the student learning outcomes. We intend to close the loop with Program Learning Outcomes as well, as one of the tests is administered during the final weeks of the students residence at Sacramento State.
Key Words
Measuring Outcomes, Use of Technology to Support Assessment
Effective Use of Indirect Assessment Methods in a Multidisciplinary Engineering Program
Steven Chin, Rowan University
Dianne Dorland, Rowan University
Olin Hall O259
Session Description
All four engineering programs at Rowan University (established in 1995), achieved ABET accreditation in 2000. Accreditation was retroactively applied to the first graduating Class of 2000. The College's multidisciplinary focus allows efficient sharing of tools and strategies, especially for indirect assessment methods such as alumni and internship/employer surveys. As neither alumni nor employer input were available during the ABET visit in Fall 2000, it was important to develop procedures for gathering and analyzing this data during the most recent ABET visit in Fall 2006. The discussion will focus on the tools and results from these surveys, and how they contributed to program assessment.
Key Words
Assessment Tools, Measuring Objectives, Measuring Outcomes, Qualitative Methods
Effective Assessment Process
Mahmoud Manzoul, Jackson State University
Tadesse Ghirmai, Jackson State University
Olin Hall O267
Session Description
Twelve program outcomes including the basic ABET outcomes a through k are adopted for the computer engineering program at Jackson State University. The assessment process starts by selecting few required courses where all the outcomes can be assessed. Each instructor of the selected courses completes a Faculty Course Assessment Report (FCAR). The FCAR includes the instructor's assessment for the course outcomes and the appropriate program outcomes. The assessment result of each program outcome is represented by an [EPAN] vector where E, P, A, and N stand for Exemplary, Proficient, Apprentice, and Novice, respectively. The EPAN vector is used as the primary method for direct assessment of students’ performance. For each program outcome, all the relevant [EPAN] vectors from the different courses are measured against a predetermined target. Consequently an appropriate action is recommended and implemented. The assessment process went through two cycles and both results will be presented. Some outcomes showed increase in student learning in the second cycle due to the data-based improvement.
Key Words
Assessment Tools, Faculty Involvement, Measuring Outcomes, Results for Improvements
|
April 13, Friday 10:10 - 11:00 a.m. |
Defining and Measuring the Meta-Learning Objectives of Multi-Disciplinary, Undergraduate Team-Based Educational Programs
Daniel Ferguson, Illinois Institute of Technology
Sherine George, Illinois Institute of Technology
Margaret Huyck, Illinois Institute of Technology
Olin Hall O157
Session Description
The project-based IPRO program has the meta-objectives of strengthening multidisciplinary teamwork, improving communication skills, teaching project management, and strengthening ethical behavior. Learning objectives have been stated in planning documents since the program’s inception 11 years ago. Determining the best way(s) to assess whether these objectives are being met has been a constant challenge. Several different evaluation tools were set up to measure attainment of these learning objectives. Unfortunately after analysis these measurement tools did not agree on what had been achieved. Therefore we completely restated the definitions of measurable skills and behaviors that we expect for each learning objective based on the assessment scale used by Purdue’s EPICS program. Each learning objective now has a set of subdomains defined as related groups of skills and behaviors [rubrics]. These rubrics are obtained wherever possible from nationally approved standard expectations. This paper will discuss the process of restatement of learning objectives and the rationale behind the process, and present data from several semesters about student self-assessments of competence.
Key Words
Assessment Tools, Measuring Objectives, Measuring Outcomes, Results for Improvements
Closing the Assessment Loop in the Small College
Frances Bailie, Iona College
Dr. Catherine Ricardo, Iona College
Olin Hall O159
Session Description
This session will focus on the problems that can arise in formulating and implementing an effective assessmemt system in a small college and how they can be addressed. Without administrative assistance from an office of assessment, the burden lies squarely on the faculty. To prepare for ABET accreditation, Iona College's Computer Science Department developed an assessment plan involving the faculty that provided valuable information for improving student learning. We also reached out to an Advisory Board of faculty, alumni, industry leaders and faculty from local universities to provide external feedback. An Assessment Committee developed several tools to measure course and program objectives by various constituencies -- faculty, students, graduating seniors, alumni and employers. Our presentation will give an overview of our assessment system, provide examples of the instruments used, explain our problem tracking system and demonstrate how our feedback procedures have successfully addressed problems reported by students, graduating seniors, alumni and our Advisory Board. We will also present our plans for an automated assessment system.
Key Words
Advisory Boards, Assessment Tools, Measuring Objectives, Results for Improvements
A Comprehensive Project- Based Program Outcome Assessment Plan:Closing the Feed Back Loop
Amir Javaheri, Virgina State University
Nasser Ghariban, Virgina State University
Jahangir Ansari, Virgina State University
Olin Hall O167
Session Description
While there are many tools available for program outcomes assessment, few of them are comprehensive or discuss how to use the assessment results to improve program quality. This presentation describes an innovative model for assessing program outcomes and discusses its application and results in Virginia State University’s manufacturing engineering program. The model uses both indirect and direct measures and is based on multiple indicators such as presentations, student portfolios, team portfolios, and report writing evaluated by different groups. The results are mapped to the program objectives and course outcomes using an innovative matrix. That matrix maintains a strong alignment to and a common frame of reference (the criteria) with the evaluation; the interpretation of results identifies the areas for improvement and prepares an improvement plan.
Key Words
Assessment Tools, Measuring Outcomes, Reporting Results, Results for Improvements
Confounded Assessment: When Direct Measures Don't Measure Outcomes
David Vaccari, Stevens Institute of Technology
Olin Hall O169
Session Description
Direct assessment of program outcomes and criterion 3 a-k outcomes becomes confounded when the student work at the base of the measures correlate to multiple outcomes. This can occur in several ways: Some assessment methods have been proposed in which each item of student work is explicitly correlated with multiple program outcomes. Similarly, some programs have developed their program outcomes so that each program outcome is correlated with multiple criterion 3 a-k outcomes. This latter situation occurs if there are fewer than eleven program outcomes. The consequence for all of these situations is that it is possible for one or more criterion 3 a-k outcomes not to be met, yet this failure could be masked by the effects of student performance in other outcomes. This presentation develops a spreadsheet model that demonstrates this problem and makes recommendations on how direct assessment should be conducted to avoid it.
Key Words
Assessment Tools, Measuring Outcomes, Quantitative Methods, Other: Direct measures
Measuring Program Outcomes using RUBRICs
Sherif Elfass, University of Nevada, Reno
Emmanuel Maragakis, University of Nevada, Reno
Olin Hall O169
Session Description
In order to facilitate the assessment of Civil Engineering (CE) Program Learning Outcomes at the University of Nevada, Reno (UNR), the CE faculty along with UNR College of Education has formulated measurable performance criteria, specifically Learning Outcome Scales (or Rubrics). The rubrics were designed to quantify the learning outcomes of the CE program. All of the classes contributing to the learning outcomes have been identified and a course matrix was developed. The process of scoring the rubric consists of two steps. Each faculty who is teaching a class identified in the course matrix evaluates students' performance and assigns a score representing the mean class performance for each sub-scale identified in the rubric. Subsequently, all the classes contributing to each sub-scale/outcome are grouped and a weighted average score, with respect to number of students in each class, is calculated. Therefore, a single score is assigned to each of the learning outcomes which is then converted to a percentage consistent with the program’s Target Performance Goal. Finally, the average performance of each outcome is evaulated by the department ABET Committee and an Annual Summary Report is generated providing feedback and recommendations to faculty for continuous improvement. A computer program was developed to facilitate the collection and compilation of data over the years.
Key Words
Measuring Outcomes, Quantitative Methods, Use of Technology to Support Assessment
Combining Technology and Prescriptive Reporting to Improve Program Outcomes
Darlena Jones, Educational Benchmarking (EBI)
Olin Hall O259
Session Description
Understanding the areas to focus attention to improve engineering programs is crucial to improving learning outcomes for engineering students. Combining that knowledge with state-of-the-art online data collection, reporting, and report distribution technologies provides college administrators a complete assessment that allows them to best understand their student / alumni / employer populations. Educational Benchmarking (EBI) performs annual national benchmarking studies in undergraduate engineering education, focusing on alumni of engineering programs, and employers of engineering program graduates. Since 2000, we have collected and reported data, in support of ABET accreditation standards, for nearly 100 engineering programs. We have developed our own online data collection, reporting, and distribution system to support higher education assessment needs. Come to this session to understand national trends in engineering education and how your program can be enhanced through new online technologies.
Key Words
Quantitative Methods, Reporting Results, Results for Improvements, Use of Technology to Support Assessment
Closing the Loop: Experience at the College of Engineering at UAE University
Reyadh Almehaideb, United Arab Emirates University
Aly S. Nazmy, United Arab Emirates University
Olin Hall O267
Session Description
This session will describe the results of two assessment cycles in the College of Engineering at UAE University. Nine different assessment tools were applied according to a pre-specified schedule. Recommendations from results of the first cycle were mostly related to improving the process itself. Simplifications were introduced to the process to encourage all faculty to get involved. Assessment of program outcomes was made through assessment of course outcomes when seeking student input. Results of the 2nd cycle resulted in improvements in the curricula and delivery method of several programs, including the general college requirements. Specific examples will be given in this session on such changes. We are now in the 3rd cycle, and results are not yet complete. At the end of the 2nd cycle, a review of program objectives was also done by including all constituencies in an open house for each program. Old and revised sets of objectives will be presented.
Key Words
Assessment Tools, Faculty Involvement, Results for Improvements, Other: Revising Objectives
|
April 13, Friday 11:10 a.m. - 12:00 p.m. |
Use of State Dept of Labor Statistics
T. Bart Quimby, University of Alaska Anchorage
Olin Hall O157
Session Description
In some states it is possible to obtain information on the occupations, salary ranges, geographical distribution, and employers of graduates without identifying specific individuals. This session will discuss the use of this data to provide a view of the accomplishment of objectives relating to preparation of students to enter the profession. This data can also be used to help focus the use of other objective assessment tools.
Key Words
Assessment Tools, Measuring Objectives, Qualitative Methods
Instructional Systems Design Based on Socratic Taxonomy
Mysore Narayanan, Miami University
Olin Hall O159
Session Description
Instructional Systems Design (ISD) Models provide structured outlines for creating a workshop, program, course or curriculum. This involves a systematic approach and a logical application of proven methodologies that can lead to acceptable solutions. ISD was made popular by Walter Dick and Lou Carey whose famous quote is: "You can't provide a solution until you know what the problem is." Several models have been proposed and all center around three major activities: Analysis, Development and Evaluation. Some of the proposed ISD models are: Dick & Carey Model, Kemp Model, ICARE Model, ASSURE Model and the ADDIE (Analysis, Design, Development, Implementation and Evaluation) Model. (Dick & Cary, 1996). Furthermore, it is well known that outcome assessment and continuous curriculum improvement now have come to the forefront of engineering education. (Rogers, 2000). In this presentation, the author tries to link the ADDIE model to Richard Paul's taxonomy of Socratic Questions. He provides guidelines for generating a set of rubrics that can help the instructors in developing certain assessment techniques.
Key Words
Assessment Tools, Measuring Outcomes, Reporting Results, Results for Improvements
Course Evaluation: Mapping Course Outcomes to Course Requirements
Amir Javaheri, Virgina State University
Olin Hall O167
Session Description
While the need for change and course improvement may be clear, there are usually great difficulties regarding what to track first. In the recent decade, there have been numerous researches about course assessment processes especially in introducing various assessmen tools. Less has been done, however, in defining how to use the assessment results for course evaluation. Improvements are usually a reflection of the instructors and not directly and systematically based on the assessment results. This presentation introduces a matrix to map course requirements to course outcomes, their assessment results and their correlation to each other. The matrix brings a strong alignment and a common frame of reference (the criteria) used for evaluation or the interpretation of results which identifies the areas for improvement and prepares an improvement plan. Application of this method will be discussed in detail in two examples.
Key Words
Assessment Tools, Reporting Results, Results for Improvements
Accreditation of an Electronics Engineering Technology Program - Setting up Educational Objectives and Outcomes
Salame Amr, Virginia State University
Olin Hall O169
Session Description
The department of Engineering & Technology at Virginia State University offers the bachelor degree of Electronics Engineering Technology at the undergraduate level. The program is already accredited by the Accreditation Board of Engineering and Technology (ABET) based upon the old criteria. The process of reaccreditation for this program is sought through the new ABET criteria by defining program educational outcomes and setting up the educational objectives. This paper also describes several assessment tools of the accreditation process such as curriculum mapping and concept inventory. Preliminary results of these assessment tools applied to the overall outcomes-based electronics engineering technology education will be discussed.
Key Words
Assessment Tools, Faculty Involvement, Measuring Objectives, Measuring Outcomes
A Streamlined, Quantitative, Web-Based, Course and Program-Level Assessment Process
Joerg Mossbrucker, Milwaukee School of Engineering
Olin Hall O257
Session Description
The Electrical Engineering program at the Milwaukee School of Engineering has implemented a streamlined course assessment process using a web-based interface for the efficient capture of information provided by faculty. Student performance data are automatically processed using a calculation scheme that defines various performance levels. Results are efficiently displayed in a quantitative graphical format giving instant feedback to the instructor. Thresholds can be defined to measure the extent that course and program outcomes are met. Results of the course assessment process are fed into the program assessment process effectively reducing the workload of both the course coordinators and the program director. High visibility of the assessment results provides easy access for the industrial advisory committee. This process was developed to address faculty concerns that the effort and volume of data of the previous process were unsustainable.
Key Words
Assessment Tools, Embedded Assessment, Quantitative Methods, Closing the Loop
Harnessing .NET as a Framework for Assessment Data Management
Jeffry Babb, Virginia Commonwealth University
Olin Hall O259
Session Description
The Microsoft .NET framework offers many easy-to-use tools, libraries and APIs to quickly develop integrated ABET assessment data management applications. The Program Assessment Reporting System (PARS) is a system under development at Virginia Commonwealth University which utilizes the .NET framwork to combine reporting, web-based faculty tools and student profile management towards a long-term ABET assessment process management tool. PARS utilizes ASP.NET web applications, web services, Microsoft Active Directory Services, XML and SQL Server 2005 towards bringing together alumni, professional community, embedded measures, FCARS, and student activities into a framework for assessment information management. Automated features link syllabi and course content to measurement efforts such that an evidence trail towards closing the loop is possible.
Key Words
Assessment Tools, Faculty Involvement, Reporting Results, Use of Technology to Support Assessment
An Advisory Council as an Effective Assessment Tool
Deborah Dunn, Stephen F. Austin State University
Michael Pickard, Stephen F. Austin State University
Olin Hall O259
Session Description
One of the most important assessment tools that we use is our Advisory Council. This assessment tool has been in place at our institution for over thirty years and is one of our most important sources of information about our performance. We collect data on all aspects of our program, including the quality of our graduates (especially compared to the graduates of competing programs); the suitability of graduates for employment; desired knowledge and skills for graduates to possess in order to be effective in current work environments; and industry trends that potentially affect curricula. This session will provide information about how we prepare for and conduct the Advisory Council Meeting, including a sample of our "typical agenda". We will also discuss how we collect and use the data gathered during the Advisory Council Meeting.
Key Words
Advisory Boards, Faculty Involvement
|
April 14, Saturday 9:30 - 10:20 a.m. |
Efforts Leading to a Successful ABET Visit: A Small Program from a Small Private College
Thomas Keyser, Western New England College
Eric Haffner, Western New England College
Olin Hall O157
Session Description
This session will describe the recent successful efforts of the Industrial Engineering Program at Western New England College and how the program was able to reasonably manage its efforts while still achieving ABET accreditation. Efforts leading to accreditation are often described as burdensome. For small programs (4 faculty) at small institutions, this is magnified, as there are fewer hands to share the load. Making assessment efforts manageable while acquiring and utilizing meaningful assessment data is challenging. This session focuses on four areas in which the IE Program was successful: 1) the direct and indirect measures used to assess and measure achievement of outcomes and objectives; 2) using the advisory board in the assessment and accreditation process; 3) obtaining meaningful assessment data quickly and using the data in the assessment process to make improvements in the educational program; and 4) the future use of technology to minimize the outcome assessment efforts.
Key Words
Advisory Boards, Assessment Tools, Measuring Outcomes, Use of Technology to Support Assessment
How Do We Know What They Really Know?
Abhinav Pamulaparthy, Illinois Institute of Technology
Daniel M. Ferguson, Illinois Institute of Technology
Olin Hall O159
Session Description
The relationship between cognitive knowledge and performance is often unclear. While traditional courses emphasize cognitive knowledge, project courses often stress performance. Undergraduate multidisciplinary project teams like the Interprofessional Project (IPRO) program at IIT often try to combine these emphases. We have developed performance measures of four major learning objectives (teamwork, project management, communication, ethical awareness). We will share the process of developing cognitive knowledge tests that presumably assess the same domains, and that are psychometrically sound. Defining the knowledge base for the learning objectives has proven to be a daunting challenge. This developmental process has evolved over two years, with approximately 70 teams averaging 12 students. Our experience illustrates the complexity of actually assessing how learning cognitive knowledge contributes to enhanced performance. We invite participants to contribute their own examples of trying to assess both cognitive knowledge and performance.
Key Words
Assessment Tools, Measuring Outcomes, Results for Improvements
Study of Student Achievement of Outcomes Using Rubrics
Swaminathan Balachandran, University of Wisconsin - Platteville
Jill Clough, University of Wisconsin - Platteville
Pat Jinkins, University of Wisconsin - Platteville
Justin Kile, University of Wisconsin - Platteville
Olin Hall O167
Session Description
The presentation will focus on the current and future use of rubrics developed by the author for collecting direct assessment data regarding the degree to which students achieved specific outcomes. First of all an overview of the assessment and evaluation manual developed by the author for the industrial engineering program at UW-Platteville will be provided. Then rubrics for measuring student achievement of outcomes of the industrial engineering program will be discussed and presented. Finally, several options available for graphical and statistical analysis of assessment data will be listed, explained, and displayed. The presentation will include an outline of the future longitudinal study of student development in achieving the outcomes.
Key Words
Measuring Outcomes, Qualitative Methods, Quantitative Methods, Reporting Results
A Web-Based Course Assessment Information System
Nicholas Flann, Utah State University
Olin Hall O169
Session Description
Our presentation will describe USU's CS Department's experience in developing and using an innovative web-based information system for supporting multiple assessment activities, including: surveying of current students, alumni, and faculty; capturing and managing assessment information for teaching portfolios, courses, and degree specializations; analyzing data across time, individual faculty, instructional styles, courses, course dependencies, and degree curriculum; and reporting results. This web-based information includes: features for capturing degree options and their general objectives; courses and their objectives; information about sections with a specific objective; tools for generating standardized syllabi; forms for course and section self assessments; and reports for summarizing and cross-tabulating the data from multiple perspectives. The system is development in ASP.NET using Microsoft SQLServer for the database.
Key Words
Assessment Tools, Faculty Involvement, Quantitative Methods, Use of Technology to Support Assessment
Designing a Student-Centered Assessment Plan for Outcomes and Objectives
Michael Misovich, Hope College
Miguel Abrahantes, Hope College
Jeff Brown, Hope College
John Krupczak, Hope College
Roger Veldman, Hope College
Olin Hall O257
Session Description
Typical assessment plans are organized in an outcome-centered manner. This may make it difficult to draw connections between student outcomes and graduate objectives. The engineering program at Hope College is designing and implementing an assessment plan which is student-centered. It is anticipated that this plan will minimize or eliminate some aspects of student outcome measurements which result in loss of validity when drawing inferences about graduates. At the same time, it will continue to provide feedback for program and course improvement without lengthy time lags. Some preliminary results will be given to illustrate these characteristics, and future enhancements allowing correlation of student outcome measurements and graduate objectives will be discussed.
Key Words
Measuring Objectives, Measuring Outcomes, Reporting Results, Results for Improvements
Direct Assessment of Criterion 3 Outcomes
Neelam Soundarajan, Ohio State University
Olin Hall O259
Session Description
During the last three or four years, the importance of direct assessment of program outcomes has been widely recognized. In this session, I will describe a set of assessment mechanisms that we have introduced in the CSE program at Ohio State that enables us to perform direct assessment of all Criterion 3 outcomes in a very effective manner. The results of the these assessments, as I will describe, have enabled us to implement numerous program improvements.
Key Words
Assessment Tools, Measuring Outcomes, Results for Improvements