WASC Proposal

WASC UGED

Proposal

Timeline

Data

Recommendations

This is a draft of the proposal that will be sent to David Goodstein for inclusion in the overall WASC proposal from Caltech. Our part should be between 2 and 3 pages. This text can be editted by logging on to the wiki and clicking the edit button.

Updates

• 24 Aug 06: updated proposal to align with WASC Handbook, pages 39-40. Comments regarding linkages to the handbook are in green; changes from the previous version are in blue.
• 30 Apr 06: cut down to two topics and focused those topics a bit. This is the final version that was used as input to the overall WASC Proposal.
• 18 Apr 06: added third topic (mentoring) plus various edits based on CCSC meeting. Changes are marked in blue, comments are marked in green.
• 16 Apr 06: first draft

---

Undergraduate Education

Andrea Dubin (ARC), Marianne Bronner-Fraser (BIO), Jean Ensminger (HSS), Csilla Felsen (ARC),
Brad Filippone (PMA), Nate Lewis (CCE), Richard Murray (chair), Niles Pierce (EAS), Josephine Suh (ARC)

Overview of Caltech Undergraduate Educational Program

Institutional context: A Statement of Institutional Context that briefly describes the institution’s background, its current context, a brief and basic presentation of its major strengths and challenges, and a synopsis of responses to issues raised by the Commission action letter and, where relevant, issues identified by the Substantive Change or Interim Report Committees. This Statement should also connect the context for the accreditation review to the institution’s vision and strategic plan.

Caltech offers a four-year undergraduate course with "options" (majors) available in applied and computational mathematics; applied physics; astrophysics; biology; business economics and management; chemical engineering; chemistry; computer science; economics; electrical engineering; engineering and applied science; English; geobiology; geochemistry; geology; geophysics; history; history and philosophy of science; independent studies; mathematics; mechanical engineering; philosophy; physics; planetary science; and social science. Each leads to the degree of Bachelor of Science.

Option requirements are established by the faculty responsible for the given degree program and consist of a set of required courses as well as recommended couses. Each course at the Institute is assigned a number of units corresponding to the total number of hours per week devoted to that subject, including classwork, laboratory, and the normal outside preparation. The Institute requirement for a Bachelor of Science degree is 486 units, which corresponds to approximately 40 units per quarter for a period of four years.

All options require students to take courses in biology (6 units), chemistry (24 units), humanities (54 units), mathematics (45 units), physics (45 units), and the social sciences (54 units). These courses form the "core curriculum", which is required of all students regardless of their degree. Course work is rigorous and students are encouraged to participate in research. The undergraduate program is thus designed to provide an intensive exposure to a wide spectrum of intellectual pursuits.

In the first year of study, students are assigned faculty advisors based on their very rough areas of interest (engineering, physics, etc). Near the end of the first year, students select an option, and during the second year they begin to specialize. Each student is then assigned a faculty advisor within the option that they select. This is normally done without regard to a students special interests (beyond the option specialization), but students can request a change of advisor if they desire. Although the average advising load across Caltech is approximately 3-4 students per faculty member, in some of the more popular options the advising load can be as high as 10-12 students per faculty member (excluding freshmen advisees).

Caltech also encourages a reasonable participation in extracurricular activities, which are largely managed by the students themselves. Three terms of physical education are required, and intercollegiate and intramural sports are encouraged. In short, every effort is made to provide undergraduate students with well-rounded, integrated programs that will not only give them sound training in their professional fields, but that will also develop character, intellectual breadth, and physical well-being.

Teaching assignments and course evaluations are the responsibility of the individual Divisions at Caltech. Most Divisions delegate the responsibility for teaching assignments to the Executive Officer for each option, who is the faculty member responsible for the undergraduate (and also graduate) degree program in a given area. Teaching evaluations are handed out in all undergraduate courses and are returned to the Division Chair, who is responsible for faculty evaluation and raises. A new online course survey is currently being developed for courses that are part of the core curriclum, which provides more detailed feedback to instructors.

The 1998 WASC review identified several issues that directly affect the undergraduate program. The issue of diversity (of both the faculty and the student body) was identified as a challenge and this continues to present a challenge for Caltech. After an initial increase, the number of women in the undergraduate body has flattened out at between 30 adn 40%, depending on the year. The number of underrepresented minorities in the undergraduates remains very low and is a great area of concern for the faculty. There has been significant progress in the recruiting and retention of women and minority faculty and this provides a somewhat better situation for recruiting students. More generally, the 1998 WASC Visiting Committee encouraged Caltech to continue its active recruiting through faculty involvement and competitive financial aid packages. The previous review also noted the desireability of providing more outlets for students interested in performing arts. . Finally, the Visiting Committee suggested that we consider revision of the Humanities and Social Sciences core curriculum.

Expected Outcomes of the Accreditation Review Process

Proposal development process: A description of how representatives of the institution’s constituencies were involved in the development and internal approval of the Proposal and how they will be involved subsequently in the review process. It is expected that key institutional leadership, especially the faculty leadership, have been involved in the design of the proposal, the format of review, the selection of special themes, the method of inquiry, and the most significant indicators that will be developed as evidence.

A committee of 6 faculty and 3 undergraduates was established to develop our proposal for self-study in undergraduate education. Richard Murray, a former Division Chair (dean) of Engineering and Applied Science at Caltech, serves as the chair of the committee and its faculty representatives have been recognized by both the students and the administration for their teaching abilities and committment to undergraduate education. The undergraduates on the committee are members of the Academic Research Committee (ARC), a committee established by the student body to help improve education and research opportunities for Caltech. ARC is also responsible for organizing the bi-annual Student Faculty Conference, which will be an integral part of the data collection and decision making process. The committee met several times during the course of the academic year to discuss the topics of most interest to students and faculty and the chair of the committee met with the Undergraduate Academic Standards and Honors (UASH) committee, the Core Curriculum Steering Committee (CCSC) and the undergraduate deans.

Based on discussions over the course of 6 months, the committee identified many potential areas for study. To help insure that we are able to make positive progress, we have identified two specific areas where we plan to focus our efforts: teaching quality and undergraduate workload. In each of these areas, we feel that a thorough assessment of Caltech's educational program would be timely and could lead to constructive changes in the undergraduate program. The recommendations of the committee were discussed with the chair of the Council on Undergraduate Education (CUE) and the WASC Steering Committee and their feedback was incorporated into the final proposal.

Expected outcomes: A description of the expected outcomes of the Accreditation Review Process. As the Commission has attempted to model this process on page 36, expected outcomes for the institution should be more than a statement of activities or identification of topics for review. The institution is expected to identify specific outcomes that it seeks to achieve through the accreditation review process and particular processes, policies, or procedures (e.g. planning, program review, establishing academic standards, faculty development, etc.) that it intends to improve through the process.

Teaching Quality

While there are many excellent teachers at Caltech, there are also examples of courses that are poorly taught and instructors who could do a much better job in their teaching performance. At the present time, there are relatively few resources available for Caltech faculty who want to improve their teaching and limited feedback mechanisms for improving the teaching effectiveness. In our initial discussions, we have identified several aspects of teaching quality that we believe should be further studied:

Course Feedback - How do faculty get feedback on their courses and whether their have achieved their desired learning outcomes, both during the term and after the term? Can course surveys, web sites, student lunches and other mechanisms be used to provide instructors more information about whether the students are understanding the material and whether the teaching methods are effective?

Teacher Training - How can Caltech usefully provide information to teachers about best practices for effective teaching, available resources and technologies, and integration of the honor system into their courses?

Faculty Incentives - Traditional teaching incentives (awards, salary increases) typically reward the best teachers but may not influence the bulk distribution of teaching. Are there incentives that can be provided to increase the average quality of teaching across the campus? Are there disincentives that can be provided to motivate those faculty that intentionally neglect their teaching responsibilities? How can faculty be assigned to the types of courses that best fit their style and abilities?

Learning Outcomes - How might Caltech explore new methods of teaching, such as active learning, lecture-less classes and project-based courses, that improve our teaching effectiveness and enhance specific learning outcomes? What new technologies might we consider for incorporation into courses and how do we provide information about these technologies to interested faculty.

Specific Outcomes: Based on our discussions, we have identified the following specific outcomes from the accreditation review process:

• Establish new processes and tools for providing feedback to instructors to improve their teaching effectiveness and enhance learning outcomes.
• Recommend and implement incentive structures that reward effective teaching.
• Identify and establish one or more new mechanisms for teaching methods to be disseminated to Caltech faculty. Closely tie these methods to specific learning outcomes that are identified by the faculty and students.

Undergraduate Workload

Another common area of concern at Caltech is the workload of the undergraduates. On the one hand, many Caltech courses are notorious for requiring more work than the number of units would indicate. At the same time, some Caltech undergraduates take a large number of units each term, creating a situation in which they have little time to integrate the material they are learning or pursue activities outside of class (including research opportunities at Caltech). Some possible aspects for further study are:

Course Underuniting - how can the number of units required for a course be accurately determined and assigned to each course? Will students be able to finish their graduation requirements if the units are increased appropriately?

Work Schedule - the timing of homework, labs, exams and courses can often lead to situations in which students must choose between completing their work and attending classes and labs. Can methods for resolving these timing conflicts be identified and implemented?

Class Attendance - in some courses, the attendance in class is lower than 50%. Are there reasons for this lack of attendance and should lectures be restructured to encourage greater attendance?

Student Health - how is the Caltech workload affecting student health (mental and physical)? Are there changes that could be made that would provide less stress on students while maintaining Caltech's rigorous educational program?

Specific Outcomes: Based on our discussions, we have identified the following specific outcomes from the accreditation review process:

• Identify root causes for the high undergraduate workload of many Caltech students.
• Based on these root causes, implement new methods for providing advice to students as well as improved processes for scheduling academic work that address the issues listed above.
• Implement processes to monitor undergraduate workload on a continuing basis to allow periodic evaluation of student workload and refinement of advising, scheduling and student health programs.

Proposed Approach

Staging of reviews: A description of how the Preparatory and Educational Effectiveness Reviews will be staged and timed to address institutional goals and to demonstrate the institution’s fulfillment of the two Core Commitments.

The two topics above represent important areas for the undergraduate educational program at Caltech and are deserving of further study. To help focus our efforts, we plan to collect substantial data that helps identify those areas where further study and changes would have the most impact. In particular, it is important to recognize that many aspects of Caltech's educational program work very well and we want to be careful not to make changes for the sake of change, but rather to be driven by a thorough understanding of what problems we are trying to solve and what metrics we will use to judge success over time.

We plan to conduct our self study in four phases:

• Phase I (Oct 06 to Mar 07) - Data Collection
• Phase II (Apr 07 to Dec 07) - Campus Discussion
• Phase III (Jan 08 to Jun 08) - Findings and Recommendations
• Phase IV (Oct 07 to Jun 08) - Initial Implementation and assessment

Preparatory Review: For the Preparatory Review, a brief description how the institution intends to present evidence to demonstrate compliance with Commission Standards including the proposed format of presentation and types of key indicators that will be included in the Institutional Portfolio. This statement should also include commentary on the effectiveness of the institution’s data gathering and analysis systems, and the steps to be taken to develop institutional evidence to support both the Preparatory and Educational Effectiveness Reviews.

The preparatory review will occur at the end of Phases I-III. We will present summaries of the data gathered during Phase I of our self study, the analysis of that data through Phases II, and our recommendations for improvements developed in Phase III. The data will include the results of surveys of teaching effectiveness and undergraduate workload, with key indicators as listed above in the description of the individual topics. As discussed in more detail below, we will make use of a variety of data gathering and analysis systems already available at Caltech, as well as developing and implementing new ideas for collecting data and providing feedback to students and instructors. We anticipate that our initial recommendations will include new processes, incentives and analysis tools that can be implemented and tested during the Educational Effectiveness Reviews.

Educational Effectiveness Review: For the Educational Effectiveness Review, a brief description of the format to be used by the institution to engage issues of Educational Effectiveness (see p. 45), the special emphases being proposed in light of the format selected, the methods to be used to engage these issues, and the ways in which the institution will address issues of student learning. This statement should also identify the key evidentiary indicators the institution will develop and/or apply to support its Educational Effectiveness Review.

The Educational Effectiveness Review will occur at the end of Phase IV and will use the "Special Themes" model (with undergraduate education as one of the three themes). Our goal is to implement the recommendations from Phase III that address the specific outcomes that we hope to obtain through the accreditation review process. Those recommendations are expected to identify specific indicators by which we can measure our progress.

The remainder of this session gives additional detail on the phases of the review and their linkage to the Prepatory Review and the Educational Effectiveness Review

Data Collection

We plan to spend the first part of the study period collecting substantial data regarding the current state of undergraduate education at Caltech as well as benchmarking against other universities with missions similar to Caltech. These activities will be undertaken by the WASC undergraduate education committee, which consists of faculty from a variety of departments as well as undergraduate students. Our current plan is to carry out these activities in the fall and winter quarters of the 2006-07 academic year, which will lead into the student/faculty conference planned for April 2007 (described in the next section).

We plan to use the following mechanisms for collecting data:

Class surveys - Caltech surveys incoming freshman and outgoing seniors each year. The surveys for 2006 have already been updated to include questions on some of the issues raised above and we can add additional questions to the 2007 and 2008 surveys as the self study progresses.

Course surveys - In addition to surveys of an entire undergraduate class, Caltech also has course evaluations for individual courses. A new online course evaluation is being used for the core curriculum and this can be used to collect data regarding to those specific courses. Data from both the online surveys and the paper surveys will be available to the committee. We will use our core curriculum course to evaluate the development and assessment of learning outcomes that are tied to Institutional objectives.

Requirements analysis - To understand student workload issues, course scheduling and related issues, we will work with the registrar's office to analyze the courses taken by Caltech undergraduates over the past several years.

Benchmarking - We plan to investigate techniques used at other schools to provide benchmarks for our analysis. We plan to identify 2-3 schools for doing a detailed benchmarking study. Candidate schools include MIT, Harvey Mudd College and Stanford.

Existing data - Caltech employs several mechanisms for feedback on its education and research program, including ABET accreditation for its engineering curriculum, external visiting committees for each Division, and bi-annual student faculty conferences. In many cases, information on undergraduate education has been collected and discussed in these forums and we will make use of this collective wisdom.

Campus Discussion

The second phase of our self study will be to discuss the data that we have collected with students and faculty, and decide on what specific topics to pursue for improving the quality of undergraduate education at Caltech. This discussion will begin with the bi-annual Student/Faculty Conference, currently scheduled for April 2007. This conference is organized by the students and we intend to form a committee on undergraduate education whose goals will align closely with those of the WASC undergraduate education committee. The output of the Student/Faculty conference is a report that includes recommendations of areas for further improvement, which will be incorporated into our self-study findings and recommendations.

In addition to the Student/Faculty Conference, we intend to use the second phase of the self study to discuss our data analysis and preliminary findings with several other groups on campus. These include:

• Academic Policies Committee - The Academic Policies Committee is responsible for making a continuous study of the Institute's academic policies. It is not be limited in any way concerning the subjects that it may take under consideration for discussion and recommendation to the Faculty Board.

• Academics and Research Committeee - This undergraduate committee serves as an objective liason between students and faculty, to facilitate effective communication, and improve the quality of learning at Caltech.

• Core Curriculum Steering Committee (CCSC) - The CCSC coordinates and supervises content and teaching of the Core Curriculum. The Committee also monitors the performance of the Core Curriculum courses and devises improvements in the core. Any proposed change in Core Curriculum courses must be reviewed and approved by the the CCSC before it is considered by the Curriculum Committee and the Faculty Board.

• Curriculum Committee - The Curriculum Comittee is responsible for general supervision over the undergraduate curriculum. All proposed changes in the undergraduate program and all proposed changes in courses, including the presentation of new courses that undergraduate students will normally take, are considered by this committee for recommendation to the Faculty Board.

• Institute Academic Council (IACC) - The IACC consists of the Division Chairs of Caltech's six academic divisions along with the President and Provost. This committee is ultimately responsible for adminsitering all teaching activities at Caltech.

• Undergraduate Academic Standards and Honors (UASH) Committee - this faculty committee is responsible for awarding a variety of academic honorsand for ensuring that a consistent set of standards and rules is applied in academic matters involving undergraduates.

Findings and Recommendations

Based on the data collected in Phase I and the discussions with the various committees and campus constituencies in phase II, the final phase of our self study will be to formulate finding and recommendations. These findings and recommendations will be provided to the Caltech administration and the Faculty Board for discussion and implementation. The timing of this phase is chosen to allow ample discussion during the academic year, so that recommendations can be acted upon in the 2008-09 academic year.

Summaries of the data collected in Phase I, the analysis and discussions of the data in Phase II and the recommendations developed in Phase III will serve as the basis for our Prepatory Review

Initial Implementation

Initial implementation of recommendations generated by the committee will occur in the 2008-2009 academic year and can be assessed at the end of that year to measure whether they are making progress.

Evaluation of the new processes that we implement---including the data collected and the improvement in key indicators---will be used as the basis for our Educational Effectiveness Review.

---

The text below this point is not part of the formal proposal. It is included here as background information on other issues that came up and were left out of the final proposal

Recommandations from 1998 WASC review

• The Visiting Committee notes the progress in recruiting and retaining femail graduate students but finds that progress is not as positive with underrepresented minorities, and suggests that Caltech continue to develop strategies to address this circumstance.
• The Visiting Committee suggests that Catlech provide adequiate support in English as a Second Language to improve the ability of graduate students working with undergraduates to communicate adequately in English.
• The Visiting Committee suggests that Caltech continue its efforts to track its graduates, perhaps merging databases with the Alumni Office, in order to address this issue [effectiveness of services for grauate students seeking nontraditional jobs] more compreshensively and systematically.
• The Visiting Committee notes that the Institute is commited to sustaning this program [SURF], and suggestions that efforts be continued to increase the endowment of the SURF program.
• The Visiting Committee recommends that students more regularly find ways to explore and clarify how the Code should apply to their relationships with each other, and speak to their desired ideals of a community.
• The Visiting Committee recommends that those most responsible for the Honor Code reach out to students at other institutions with similar responsibilities to learn more, as they will, about the social implications of Honor Codes.
• The Visiting Committee suggests that the Instiute consider the revision of the Humanities and Social Sciences Core with the same intensity of focus as it has considered the science Core.
• The Visiting committee encourages Caltech's continued diligence to recruiting and retenion (of underrepresented students) and to life on campus. [Specifically meniotns increasing diversity of the faculty.]
• The Visiting Committee encourages Caltech to continue its active recruiting. The faculty involement during the last cycle had noticeable positive results. The Financial Aid packages are being made more competitive. The Institute should continue to look for the best ways to attract and retain a high-quality and inclusive student body.
• The Visiting Committee encourages Caltech to pursue these initiatives [in arts and performing], which will add to campus cultural and intellectual life.

Unused Material

The material below was removed from the proposal to keep it concise and focused.

Teaching Quality

• Role of the Core Curriculum - The core curriculum is the dominant experience for students arriving at Caltech and establishes the common background for option-specific courses. Is the teaching in the core preparing students for the style of learning that they will encounter in later classes? Are core courses providing the material required for later courses and for students' post-Caltech education and careers? [RMM: Scott Fraser would really like us to include some evaluation of the core, not necessarily to change it, but to provide data about its relevancy. Should we include this here?]

Student Workload

• Independent Study - is the structure of the Caltech undergraduate education one that allows "inspired independent study" for those students who are interested? How might research opportunities be better integrated into the curriculum requirements. [RMM: This topic overlaps with the Undergraduate Research self-study topic.]

Student Advising

[RMM: This came up in the CCSC meeting and there was a lot of resonance with looking into this.]

The low student faculty ratio at Caltech (approximately 3:1) allows undergraduates to have a much richer interaction with their faculty advisors than might be possible at a larger school. However, there are challenges within the structure of Caltech that can lead to less than idea situations. One issue is that students are not uniformly distributed across Caltech's departments, so while the student/faculty ratio might be low in some areas, it can be quite high in others (e.g., in recent years the EE faculty have averaged 12 student advisees per faculty member, not including freshmen). In addition, the intense research focus at Caltech provides opportunities for some students to get involved a faculty lab during their studies, but some students complain that faculty to not spend adequate time on their advising activities. Some possible areas of study include:

• Advising Load - How can advising be better distributed across the entire Caltech faculty, so that we can take best advantage of the low student/faculty ratio at Caltech?

• Advisor Training - What information and training should Caltech provide to faculty advisors? How might best practices in undergraduate advising be shared between faculty?

• Advisor Assignments - How should students be matched with an advisor? Should advisor assignments remain static during in the sophomore through senior years?

• Advisor Feedback - How can students provide constructive feedback to their advisors, especially given the very individual (and hence attributable) nature of the advisor/student relationship?

Other Issues

[RMM: I want to eliminate this section. If there are topics here that are really important, we should try to include them under the three board themes above.]

In addition to these broad issues, there are a number of more specific items that we would like to collect information about and consider as possible components of the self study on undergraduate education:

• Classroom experience (across the variety of courses that students take)
• Might touch on tools (eg, MATLAB), breadth of the core [RMM: captured under core evaluation]
• Look into student health? Compare to other universities [RMM: captured under student workload]
• Perhaps link to research and honor system topics
• Course attendance, classroom experience and *learning* [RMM: captured under teaching quality]
• Can we find out why students aren't coming to class? [RMM: captured under student workload]
• Focus on what students learn?
• Content of the core [RMM: captured under core evaluation]