Expect More. Achieve More.
Direct Methods
Direct measures provide a way to compare actual student work to the intended outcomes of the program. When these are well executed, they provide the best evidence of student achievement in each area. They are, however, more time- and labor-intensive to administer than indirect measures. Direct methods may include capstone courses, course-embedded assessments, juried reviews, standardized or locally developed tests, portfolios, and pretest/posttest assessment. Each is described below, along with a suggestion of the trade-off of benefits and disadvantages that is inherent with each method.
1. Capstone Courses
Capstone courses draw upon and integrate knowledge, concepts, and skills associated with the entire curriculum of a program. Taken normally in the senior year, capstone courses ask students to demonstrate facility in the program's learning outcomes, in addition to other outcomes associated with the particular course. Within a capstone course, evidence of student learning may include comprehensive papers, portfolios, group projects, demonstrations, journals, or examinations.
But how does one use this evidence to assess the overall program? The final grade for the course, being a single measure, does not provide a separate assessment of student achievement in the various learning outcomes for the program (although achievement in each of the learning outcomes may combine into the final grade). One method of assessment in capstone courses is to evaluate student work with an eye toward the multiple dimensions of the program's outcomes. More than one faculty member can be invited to assist in the assessment of student work, e.g. in a project presentation. The assessment of a major paper or project, or set of papers or projects, may be broken down into sub-assessments of each learning outcome.
The benefits of a capstone course include the increased ability of students to integrate their learning; close association between the assessment tool and the program's particular learning outcomes; rapid feedback on the program; and association of a faculty member who contributes to program assessment through the professor's normal teaching load.
Disadvantages include disciplinary differences in the appropriateness of a capstone course in the curriculum; the time required to develop and gain approval for the course; staffing requirements; slotting the course into the curricular sequence; and normal variations in course content as the course changes hands from instructor to instructor, i.e. reliability. Also, this type of assessment does not lend itself easily to comparisons of student work early and late in their academic career.
Below is a sample Capstone Course Syllabus:
2. Course-Embedded Assessment
In course-embedded assessment, student work in designated courses is collected and assessed in relation to the program learning outcomes, not just for the course grade. As in the capstone course, the products of student work need to be considered in light of the multiple dimensions of the learning outcomes. Products may include final exams, research reports, projects, papers, and so on. The assessment may be conducted at specific points (e.g., introductory course and upper-level course) in a program.
Benefits include the fact that assessment is conducted as part of the normal workload of students and faculty members, although additional work may be needed to incorporate program assessment into the course.
Disadvantages include the potential for a faculty member to feel that her or his work in a particular course is being overseen, even if it is not. Also, rubrics may need to be chosen or developed that are associated with the particular learning outcomes, increasing initial preparation time.
3. Juried Review of Student Projects
In some disciplines students produce projects or performances that can be evaluated by a panel of experts, a jury. The jury may be composed of either internal reviewers (members of the BSU faculty) and/or external reviewers. Juried review is most common in artistic disciplines such as Theatre, Dance and Music but could be used in many fields. Juried review may be initiated as part of the teaching and learning process for individual students, but it may also be used for program-level assessment. In this case, the individual students identity is unimportant and may even be masked. For program-level assessment the focus of juried review is on the strengths and weaknesses demonstrated by all students in the program, rather than on the performance of a particular student. If this type of assessment takes place only at the end of the program, it may demonstrate mastery of learning outcomes. If it takes place early in the students academic career and again at the end, it can also show the development of student skills across the academic program.
The benefit of external juried review is that the reviewers do not know the students involved and are likely to appear to be more impartial. The benefit of internal juried review is that it takes a form of assessment that may already exist for teaching purposes and moves it to the program level at little or no additional cost. Using a mixed jury format (combination of internal and external jurists) may allow for a more balanced assessment of students performance.
The disadvantage of external juried review is the cost of bringing in the external experts. The disadvantage of internal juried review is the possibility of bias among faculty members who already know the individual students and their work well.
4. Standardized Tests
Several commercial test publishers offer standardized tests for various types of learning outcomes, such as critical thinking or mathematical problem solving. Scores on tests such as the GRE or the Massachusetts Test of Educator Licensure (MTEL) may be used as evidence of student learning. Before using a specific standardized test for program assessment, it is very important to ensure that the test measures specific knowledge and skills defined by the academic program.
Benefits include the reliability and validity of an assessment instrument that is commercially developed, eliminating the arduous process of developing an instrument in-house; simplicity in administration and evaluation of test results; and the potential for cross-institutional comparisons of results.
Disadvantages include the generic nature of standardized tests and their potential lack of fit with a particular program; a possible lack of motivation by students to take the test or do well on it; and the debatable question of whether a standardized test gives a true measure of student learning. Also, test companies charge substantial fees for these tests, which is an added administrative cost or possibly a cost to the students.
Sample Standardized Tests:
· California Critical Thinking Dispositions Inventory
· Collegiate Learning Assessment
5. Locally Developed Tests
Faculty members in a program may decide to develop a test that is reflective of the programs mission and learning outcomes. The test is usually graded by multiple evaluators. Locally developed tests are less costly than a standardized test, but require work by the programs faculty in development and scoring.
Benefits include the ability to tailor a test to a specific program and the learning process for program faculty members as they review the content and structure of the test items collaboratively.
Disadvantages include the challenge of developing a test with proven validity and reliability potential need to develop rubrics and train multiple test evaluators in the use of these rubrics, and the need to develop a new test periodically6. Portfolio Evaluation
A portfolio is a compilation of student work that, in total, demonstrates a students achievement of various learning outcomes. Portfolios can be created for a variety of purposes aside from program assessment, such as fostering reflection by students on their education, providing documentation for a students job search, or certifying a students competency.
Portfolios may be grouped into three types: formative, summative, and professional. Formative portfolios collect evidence of a students work over the entire course of the academic program, showing beginning work as well as work demonstrating mastery. For summative portfolios, only the best work in each category is selected. Professional portfolios select from among the best work and retain only those pieces that one would want to present to the public, for example for a potential employer. The different types of portfolios may be used for different purposes in program-level assessment. Formative portfolios document development over time and may be thought of as equivalent to a pre-post evaluation. Summative portfolios do not show development but may be used to demonstrate mastery of intended learning outcomes. Professional portfolios provide evidence of mastery and also relate directly to programs whose goals include employability or preparation for graduate study.
Portfolios may combine multiple types of evidence and are not necessarily limited to classroom work. For example, portfolios may contain research papers, presentations, videos, audio recordings, work done through employment, or journal entries discussing co-curricular activities or programs. Once the material is collected, it falls upon an individual or group to establish a system by which to evaluate the contents of the portfolio in terms of a programs learning outcomes.
In the College of Education and Allied Studies at BSU, portfolios are used to document each students competence in teacher preparation as required by the National Council for Accreditation of Teacher Education (NCATE). This is a different purpose from that for program assessment. In program assessment, a cross section of students may be appropriately sampled to evaluate student learning outcomes, but in teacher preparation, the intent is to validate every students competence.
A key question in portfolios arises in the collection of evidence. In teacher preparation, students themselves collect and save the material, and online systems are now available to assist in that process. But for program assessment, the department itself may have to assemble the student portfolios; in this case, issues must be considered about how the students are to be informed of the fact that their work is being assessed for programmatic reasons. Some faculty members ask students to sign consent forms to copy work products and to use student work products in accreditation reports. More information on this can be obtained from the BSU Institutional Review Board.
Benefits of portfolios include the ability to document student development over time, and the potential benefit to the students of seeing their own development and in collecting material that may support their career goals. Thus, program assessment becomes an integral part of the learning process.
Disadvantages include a labor-intensive process in the evaluation of evidence in student portfolios. Also, there is an expense in storing and organizing the evidence.
Sample rubric for portfolio assessment
7. Pre-test/Post-test Evaluation
One of the questions that come up in assessment is not only whether students can demonstrate the learning outcomes when they graduate, but how much of what they can demonstrate was actually gained during their time in the program. This suggests the need to assess the students' knowledge and skills at the point of entry into the program and, later, at the point of exiting the program. In pretest/posttest assessment, student work is assessed both early and late in their academic career, from which the growth and development of the students can be deduced.
Several of the previously-described tools lend themselves to pre-test/post-test evaluation. Formative portfolios that collect evidence throughout a student's academic career can intrinsically be a type of pre-post evaluation. Course-embedded assessment in which student work is collected from introductory and upper-level courses also provides a type of pre- and posttest evaluation, although the level of difficulty in the two courses can be expected to differ considerably. Standardized or locally developed tests can be administered at two times in a student's career to assess learning, if the second administration follows the initial administration by a period of time. For standardized test this is typically no more frequently than an annual administration, while locally developed tools may be able to be administered at the beginning and end of a single semester.
Benefits include the ability to gain insight into students' academic development.
Disadvantages include the increased amount of work involved in assessing student work more than once, and the difficulty of designing tests or assessment tools that are truly comparable at different times.
Last Modified: October 19, 2012
© Bridgewater State University, 131 Summer Street, Bridgewater, Massachusetts 02325, 508.531.1000
Contact Us
: Policies
: Privacy
: Site Map