Academics

How will you prepare to live and prosper in a world where the future is up to you? WPU offers education for the 21st century. We welcome your interest.

Overview of Assessment

Overview of Assessment

Student learning in the CHOICES program will be assessed by multiple methods including nationally-normed instruments, a university-wide ethical decision making (EDM) grading rubric, student self-report and existing data. We will use a combination of formative and summative assessment, direct and indirect measures, and pretesting and posttesting to determine the effectiveness of the CHOICES program. Table 10.1 below summarizes assessment of the Student Learning Outcomes associated with the CHOICES program.

Table 10.1: Assessment of Student Learning Outcomes

Student Learning Outcome 1: Understand concepts that provide a common language for analyzing ethical situations.

  • Assessment: Ethical Decision-Making Knowledge Pretest and Posttest
    • Assessment Type: Direct measure of student learning
    • Assessment Timeline: Pretest and posttest during the PSS 100 course

Student Learning Outcome 2: Adopt values consistent with ethical decision making.

  • Assessment: Values in Action Inventory
    • Assessment Type: Direct measure of student learning
    • Assessment Timeline: Orientation (pretest) and end of senior year (posttest)
  • Assessment: BCSSE and NSSE
    • Assessment Type: Indirect measure of student learning
    • Assessment Timeline:
    • BCSSE (pretest)- orientation of first year
    • NSSE (posttest) graduating seniors
  • Assessment: Student Conduct Issues and Faculty Survey of Plagiarism
    • Assessment Type: Indirect measures of student learning
    • Assessment Timeline: Baseline data collected in 2010-2011; summative data will be collected each year of the CHOICES program
  • Assessment: Ethical Decision-Making Grading Rubric
    • Assessment Type: Direct measure of student learning
    • Assessment Timeline: Baseline data collected at beginning of PSS 100; post-test data at the end of PSS 100 and the end of PHL 400

Student Learning Outcome 3: Apply an ethical decision-making framework when making ethical decisions.

  • Assessment: Ethical Decision-Making Grading Rubric
    • Assessment Type: Direct measure of student learning
    • Assessment Timeline: Pretest data collected at beginning of PSS 100; post-test data at the end of PSS 100 and the end of PHL 400

We have one additional assessment, the Collegiate Learning Assessment (CLA). Although the CLA does not directly measure any of the SLOs, we expect critical thinking to improve as a result of SLO 3 which involves learning to use a systematic reasoning process to evaluate ethical decisions. The CLA provides a direct measure of student learning related to critical thinking and will be administered during students' first and fourth years.

Description of Direct Measures of Assessment. Direct measures provide quantitative data, often normed, that indicate a change in student knowledge, skills, values or behavior. The CHOICES program has at least one direct measure of student learning for each student learning outcome as illustrated in Table 10.1.

Values in Action Inventory. The Values in Action Inventory (Peterson & Park 2009) measures character traits, which indicate a person's values and are displayed in one's behavior. The VIA measures 24 character traits, 6 of which will be analyzed to determine student learning in the CHOICES program. The VIA poses scenarios confronted in everyday life and respondents choose responses that are most characteristic of them. Unlike typical measures of values where people choose from a list of values they hope or believe describe them, the VIA asks respondents to report behaviors (with the values driving those behaviors implicit in the assessment), making socially-desirable answers less likely. Each of the scales and definitions shown below, as well as more information about the assessment can be found at www.viacharacter.org. The relevant subscales from the VIA, shown below, overlap considerably with definitions of our six core values.

  • Judgment and Open-Mindedness: Thinking things through and examining them from all sides; not jumping to conclusions; being able to change one's mind in light of evidence; weighing all evidence fairly.
  • Responsibility and Perseverance: Finishing what one starts; persevering in a course of action in spite of obstacles; "getting it out the door"; taking pleasure in completing tasks.
  • Honesty and Integrity: Speaking the truth; presenting oneself in a genuine way and acting in a sincere way; being without pretense.
  • Kindness and Care: Showing compassion or altruistic love; doing favors and good deeds for others; helping them; taking care of them.
  • Teamwork and Citizenship: Working well as a member of a group or team; being loyal to the group; doing one's share.
  • Fairness: Treating all people the same according to notions of fairness and justice; not letting personal feelings bias decisions about others; giving everyone a fair chance

The VIA provides two different scores. Each of the 24 character traits is scored on a scale ranging from 1-5. In addition, the VIA provides a rank ordering of the 24 character traits ranging from the individual's top strengths to their less-used strengths. After baseline data is evaluated, Peace will set a benchmark for improvement on the targeted 6 subscales shown above.

Collegiate Learning Assessment. Peace will also track progress on students' scores on the Make an Argument and Critique an Argument subscales of the Collegiate Learning Assessment (CLA). The Eight-Question Model for Ethical Decision Making taught in the CHOICES program is linked to the eight elements of thought as identified by the Foundation for Critical Thinking (www.criticalthinking.org). Development of skill in ethical decision making should enhance critical thinking skills, particularly on the Make an Argument and Critique an Argument subscales. For example, the Eight-Question Model teaches students to consider the perspective of others, to test their assumptions, to think about the possible consequences of their actions and to select information to support their arguments. These same elements form part of the scoring rubrics for the Make an Argument and Critique an Argument subscales of the CLA.

The benchmark for improvement on the CLA will be a value-added score in the "above expected" range for seniors measured in the spring of 2015. Additionally, we will expect an average increase of 10% in the score - or the equivalent of one letter grade of improvement in a class setting -- on the Make an Argument and Critique an Argument subscales over the average of scores from 2009, 2010 and 2011 Peace seniors. Although we will use the CLA to measure growth in critical thinking, the transfer of skills from one domain (ethical decision making about dilemmas faced by individuals) may not generalize to the content more common in critical thinking tasks (Weisberg, 2006) and thus scores on the CLA may not increase to the targeted benchmark.

Ethical Decision-Making Grading Rubric. Peace has adapted the Ethical Reasoning Value Rubric created by the American Association of Colleges and Universities (AACU). The AACU rubric was adapted to include all elements of our Eight-Question Model for Making Ethical Decisions. It was pilot-tested with case study analysis papers collected in PSY 320, Adult Development, during our planning year and further refined after training for PSS 100. A copy of our rubric can be found in Appendix D. Peace will measure five elements of ethical decision making skill within the rubric including

  • ethical self-awareness (value, SLO #2),
  • ethical issue recognition (skill, SLO #3),
  • ethical perspective taking (skill, SLO #3)
  • evaluation of ethical choices (skill, SLO #3), and
  • implications of ethical choices (skill, SLO #3).

Each of these five elements will be scored on a 1 (baseline) to 4 (capstone or mastery) level with specific scoring criteria given for each element at each level. We expect first-year students, on average, to score between 1.0 and 2.0 on the baseline case study for each of the five elements. The targeted benchmark performance for the end of the PSS 100 course will be an average score of 2.0. For seniors, an average target score of 3.25 will be set for each of the five elements. (See Appendix D for a copy of the Ethical Decision-Making Grading Rubric.)

Ethical Concepts Pretest and Posttest. This locally-developed assessment short answer test will measure students' understanding of the following concepts: values, integrity, character, moral temptations, ethical dilemmas, and ethical relativism. In addition, students will need to list and define each of WPU's 6 core values. The targeted average score on the posttest will be 20% higher than the pretest score. (See Appendix G for a copy of this assessment.)

Indirect Measures of Student Learning. Indirect measures of student learning include student or faculty surveys, usage data, and other forms of information that reflect program impact without directly measuring student learning.

Beginning College Student Survey (BCSSE). The BCSSE measures incoming students' academic and co-curricular experiences in high school and their expectations for college work and experiences. The BCSSE will be completed on-line during orientation of the students' first semester at WPU. More information about the BCSSE can be found at http://bcsse.iub.edu/.

National Survey of Student Engagement (NSSE). The NSSE measures students' self reported learning and experiences in college. Items on the NSSE parallel those on the BCSSE to allow institutions to ascertain student perceptions of their growth and the quality of the experiences the institution is providing. Entering students will be given the BCSSE during orientation and the NSSE at the end of the first year and senior year. More information about the NSSE can be found at http://nsse.iub.edu/. Companion items from the BCSSE and NSSE that will be tracked include:

  • had serious conversations with students of a different race or ethnicity than your own;
  • had serious conversations with students very different from you in terms of their religious beliefs, political opinions or personal values;
  • tried to better understand someone else's views by imagining how an issue looks from his or her perspective ;
  • developed a personal code of ethics (NSSE only);
  • examined the strengths and weaknesses of your own views on a topic or issue (NSSE only; and
  • solving real-world complex problems.

For items contained on both the BCSSE and NSSE, benchmark scores for the above items will be set based on a score that is .5 higher on the NSSE than the BCSSE. For items measured only on the NSSE, seniors' scores on the targeted items will be compared to an average of senior scores from 2009 and 2010 with a target improvement of .5 of a point.

Academic and Student Code of Conduct Violations. Various forms of existing data can inform us whether students are better at making ethical decisions after the CHOICES program begins. Data related to academic dishonesty and student conduct violations are maintained based on the type of violation. Based on the average number of violations over the last three years in both academic dishonesty and student conduct, we will set a target of 10% fewer violations for the first four years of the CHOICES program. As more students participate in the program over the years, violations should steadily decrease.

Faculty Survey of Academic Dishonesty. Baseline data were collected in the summer of 2010 related to the percentage of faculty members who reported cases of plagiarism, cheating and other forms of academic dishonesty. This survey data will be collected again in the summer of 2014, three years after the CHOICES program is implemented. We expect a drop of 20% in cases of academic dishonesty among students who have experienced the CHOICES program since their first year at WPU.

Formative and Summative Assessment. Data will be collected for both formative (student feedback) and summative (student learning and program evaluation) purposes. The comprehensive timeline shown in Appendix E shows what assessment will occur during each of the first five years of the QEP. Formative assessment will be used throughout the program to inform and improve student learning. All summative assessment will have some pretest or baseline data available for comparison purposes.

Cross-Sectional and Longitudinal Comparisons. Because of the small number of students enrolled in each class level at WPU, all students will be assessed on each instrument. Effectiveness of the CHOICES program will be analyzed in two ways:

  1. In the 2014-2015 academic year cross-sectional comparisons will be made among 50 randomly-selected case study analysis papers that were collected for the baseline assessment, the summative first-year assessment and the summative senior-assessment.
  2. In the spring of 2015, a longitudinal analysis will be conducted for all students who entered in 2011 and participated in assessment at all three time periods (beginning of the first year, end of the first year and end of the senior year). This longitudinal comparison will be repeated in the spring of 2016 for students who entered in the fall of 2012 and participated in assessment during all three time periods.
none