2024-2025 Graduate Certificate Program Assessment

Academic Program Assessment Report 

Academic Year(s) Assessed: 2024-2025 

College: Letters & Science

Department: Master of Science in Science Education 

Department Head: John Graves

Submitted by: John Graves 

Program(s) Assessed

Program Director: John Graves, EdD

Faculty Reviewers: Robyn Gotz, Pilar Martland, Holly Thompson-Reed 

Section 1. Past Assessment Summary.  

Response  

We were confused by the divergent assessment of the learning outcomes. One reviewer stated they were clear, measurable and needed no edits. The other indicated that some were not at a high cognitive level. Examples would be helpful.  

We did find value in the assessment as we shared the 2022-2023 results with MSSE faculty at our 2025 Spring Faculty Retreat. We used the results to inform instructors about incorporating the Next Generation Science Standards (NGSS) in their courses and provided additional information about how to include learning outcomes in course syllabi. Since not all instructors were present, we created instructional video that were shared. Those in attendance also engaged in discussions after each presentation. 

Institutional Assessment Data Request  

The two courses that meet the institutional assessment core quality learning outcomes are MSSE 501 Inquiry Through the Lens and Science and Engineering Practices and MSSE 575 Capstone Symposium. The culminating activity in MSSE 501 is a personal inquiry project where students choose a topic and engage in an inquiry investigation as a learner. They then synthesize their thinking and problem solving in the final submission, which can take the form of a narrative, a presentation, a video or other media. Since the topics are individually chosen, the address both local and global issues. Titles of project from a recent course include Coastal Erosion and Biodiversity; Voltaic Cells; Dog House Inquiry; Teaching the Impacts of Nuclear Disasters; Drought on the Blackfeet Reservation and Skeletomuscular Injuries and the Healing Process. 

The MSSE Capstone Symposium is the culmination of a 5-semester sequence of courses that results in a classroom-based or science-based research project. A professional paper of the research topic accompanies the public presentation and subsequent oral defense of the project. In addition, each student creates a virtual poster that is displayed the day of their presentation. The poster and paper are then archived in the MSU library. Examples of projects titles from summer 2025 include Land-based Learning in Junior High School; Engagement of Case Studies in the Earth Science Classroom; Nature-connectedness, Knowledge, and Attitudes of Camp Zoo Attendees and Note-taking Strategies for Middle School Science. All of the projects contain evidence of thinking and problem solving, effective communication through written and oral methods and relate learning to local and global issues. 

Section 2. Actionable Research Questions for Your Assessment  

1. Are students able to define and apply (chemistry, climate science, Earth science, elementary school science, life science, physics and STEM) concepts as described in the Next Generation Science Standards through assignment artifacts? 

2. Are instructors explicit about the expectation that students define and apply (chemistry, climate science, Earth science, elementary school science, life science, physics and STEM) concepts as described in the Next Generation Science Standards in course syllabi and assignment instructions? 

Section 3. Assessment Plan, Schedule, and Data Sources 

The previous plan was changed because we updated the learning outcomes for the graduate certificates and adjusted our assessment planning schedule. 

Our previous learning outcomes significantly varied between the different graduate certificates. For example, the climate science certificate included eight outcomes. The chemistry, Earth science, elementary school science, life science and physics graduate certificates had five outcomes that were shared. With the addition of the STEM certificate, and at the suggestion of the 2023 MSSE Program Reviewers, we pared the learning outcomes for all graduate certificates to four. 

Program Learning Outcomes for Graduate Certificates 

1. Demonstrate a mastery of discipline specific concepts, theory and application in (chemistry, climate science, Earth science, elementary school science, life science, physics and STEM) at the learner’s appropriate instructional level: elementary, middle or high school. 

2. Define and apply (chemistry, climate science, Earth science, elementary school science, life science, physics and STEM) concepts as described in the Next Generation Science Standards 

3. Identify that topics in (chemistry, climate science, Earth science, elementary school science, life science, physics and STEM) are linked together and demonstrate how (chemistry, climate science, Earth science, elementary school science, life science, physics and STEM) is connected to other science content. 

4. Plan, create and adapt teaching activities to custom (chemistry, climate science, Earth science, elementary school science, life science, physics and STEM) lesson strategies. 

To answer the question, Are students able to define and apply (chemistry, climate science, Earth science, elementary school science, life science, physics and STEM) concepts as described in the Next Generation Science Standards through assignment artifacts, a threshold of 85% of students achieving at or above the expected level was established. This reflects a strong yet attainable indicator of the learning outcome. It ensures broad-based success, promotes robust understanding of the outcome, and encourages continuous instructional improvement.  

To answer the sub-question, Are instructors explicit about the expectation that students define and apply (chemistry, chemistry, climate science, Earth science, elementary school science, life science, physics and STEM) concepts as described in the Next Generation Science Standards in course syllabi and assignment instructions, no threshold was established. Because this is a new program learning outcome, we were interested in whether instructors explicitly addressed this outcome and effectively communicated it to students.  

Section 4. What Was Done?  

For this assessment, we chose to focus on this learning outcome: 
 

Define and apply (chemistry, climate science, Earth science, elementary school science, life science, physics and STEM) concepts as described in the Next Generation Science Standards.  

Since this is a new outcome, we were especially interested in two questions: 1) are instructors explicit about including this outcome in their syllabi and/or directions that guide student assignments and 2) are students addressing this outcome in their assignments? As a result, independent assessments were conducted on the course materials created by the instructors and on student artifacts submitted.  

To support our assessment plan, we created a curriculum map (Appendix Link) to help us meet the following goals: 
 
• to provide transparency for our program learning outcomes and individual courses 
• to assist with instructional alignment 
• to assist with program assessment 

• to guide program professional development to better support faculty 

At this point the curriculum map is a work-in-progress, and we have two-year goal of completion. This supports our on-going objective of updating all MSSE course syllabi in CIM, a daunting task because we have over 90 courses with 16 different rubrics. 

1. Self-reporting Metric (required answer): Was the completed assessment consistent with the program’s assessment plan? If not, please explain the adjustments that were made. Yes.
2. At the request of the MSSE Director, all instructors of the courses used towards graduate certificates were emailed a request to submit student assignment artifacts that addressed the Next Generation Science Standard (NGSS) learning outcome. Artifacts were submitted for 19 courses. Each artifact was assessed by five reviewers using the NGSS Connections Rubric. The reviewers were graduates of the MSSE Program and were selected after the MSSE Director emailed graduates asking for volunteers to serve as reviewers. A total of nine graduates and the director reviewed the artifacts. The scores were averaged for data reporting. A total of 71 student artifacts from 19 courses were assessed. For data analysis purposes, each category is reported separately. The reviewers determined that a rating of achieving, which simply implied that NGSS concepts were addressed, was not a true representation of achievement. Artifacts included lesson plans, discussion posts, presentations, videos (Appendix A).  
3. Syllabi, assessment rubrics, and documentation of assignment instructions were requested from instructors. Instructors from 19 courses provided artifacts. The same reviewers of student artifacts also reviewed the information instructors provided. For data analysis purposes, each category is reported separately. The reviewers determined that a rating of achieving, which simply implied that NGSS concepts were addressed, was not a true representation of achievement. Artifacts from instructors included lecture notes, syllabi, rubrics and more (Appendix B). 

Section 5. What Was Learned 

The results of the NGSS Connections Rubric indicated that 7% of the student artifacts were excellent, 27% were between achieving and outstanding, 74% were at the achieving level and 20% were inadequate (N=71). An example of an outstanding artifact was when a student specifically referenced Next Generation Science Standards in a discussion post by stating, “…students come up with a more refined way to recycle…” then referencing NGSS standards: HS-ESS3-4; HS-ETS1-3; HS-LS2-7. An example of an achieving level score included the words NGSS, Next Generation Science Standards or a reference to one of the science and engineering practices or crosscutting concepts with no other explanation or detail. The threshold of 85% of assessed students scoring at or above the between achieving & outstanding level was not met (Figure 1). 

Figure 1. Student NGSS Connections Rubric Results, (N=71). 

The results of the Instructor NGSS Connections Rubric indicated that 11% of the instructors provided explicit (outstanding) descriptions to students regarding NGSS expectation in assignments. Thirty-two percent provided instructions that were between achieving and outstanding, 25% were at the implied (achieving) level and 32% did not include any at all (N=19). An example of explicit instructions one instructor provided was, “List any NGSS standards your lesson addresses.” (Figure 2). 

Figure 2. Instructor NGSS Connections Rubric Results, (N=19). 

Upon further discussion and reflection with reviewers and instructors, it should be noted that we have determined that some courses will never meet the NGSS science disciplinary core standard because they are content specific courses and NGSS disciplinary core concepts do not exist for some courses. For example, content in special relativity, quantum mechanics, paleontology and other courses are not included in NGSS. However, since NGSS is comprised of three dimensions: disciplinary core ideas, science & engineering practices and crosscutting concepts, this Program Learning Outcome can be met through the dimensions of NGSS that are appropriate to each course, especially the eight science & engineering practices and the seven crosscutting concepts. 

Summarized from NSTA.org and retrieved from http://mvsdcurriculum.monte.k12.co.us/ngss-overviewprogression.html 

It was also noted that 53% of instructors did not provide students an assessment rubric for the assignment submitted as an artifact. 

One strength observed through this assessment was that MSSE students are learning, applying and thinking more like scientists. One student reflected on the inquiry practice, “Although I’ve been teaching for nearly 17 years, I feel like it’s only been in the last three years or so that I have begun integrating more inquiry into my teaching. I’m reminded of how far I’ve come but also how far I have to go.” In a microbiology course, one student wrote as part of her conclusion, “By looking at groups of structural genes coding for LPS, rod shaped structure, two component stress response system Cpx lead me to believe that my bacteria is a G- pathogenic rod.” 

What we learned from the analysis of the data is that instructors and students are becoming more cognizant of the Next Generation Science Standards, but the program needs to assist instructors to incorporate the NGSS dimensions in their instruction. This will be an easy, yet challenging process. It’s easy because science and engineering practices and crosscutting concepts are the native language of scientists and science researchers. Providing support for instructors to explicitly add the assessment of NGSS dimensions to their syllabi and instruction is process that will take some time. 

Section 6. How We Will Respond. 

To address the results of this assessment, we propose the following plan: 

• Dedicate the Spring 2026 Instructor Retreat to this Program Learning Outcome. 

• Share these results with instructors. 

• Provide direct instruction about NGSS. This process began in 2021, but since the program was not requiring specific NGSS language in course syllabi, it fell flat. 

• Establish the expectation that NGSS language be included in syllabi. Provide training videos and electronic examples/templates for instructors. 

• Set a target date of October 2026 for all syllabi to be updated to include NGSS language. 

• In anticipation of the 2026-2027 Graduate Certificate Program Assessment, dedicate the 2027 Instructor Retreat to the establishment of a keystone assignment and assessment rubric for each course. The keystone assignment and associated rubric will be the artifacts for future program assessments. The keystone assignment will be aligned to the graduate certificates and MSSE Program Learning Outcomes. 

MSSE has internal resources to support this proposed effort. 

Section 7. Closing the Loop(s)  

We plan to close the loop as proposed in the previous section. Results of this assessment will be shared with the MSSE Steering Committee in November 2025 and with instructors at the 2026 Spring Instructor Retreat. Changes will be made based on the findings and/or faculty input from the spring retreat, which will result in measurable improvements in preparation for upcoming assessments. We expect those measurables to include, but not be limited to the inclusion of NGSS language in all course syllabi, the inclusion of a keystone assessment artifact and supporting rubrics for all courses, and the completion of the MSSE Curriculum Map by October 2027. 

An external research study helped complete the feedback loop by examining how MSSE field experiences translate into classroom practice. Commissioned by MSSE, the study employed a mixed-methods approach, gathering insights from 23 student surveys and interviews. Findings revealed that immersive, hands-on fieldwork delivers a range of interconnected benefits beyond subject matter expertise. These include increased confidence, stronger collaboration skills, deeper reflective habits, and an enhanced ability to link scientific concepts to real-world applications. 
 

Teachers consistently expressed high levels of enjoyment, a sense of novelty, and boosted confidence, suggesting that field-based learning fosters motivation and self-efficacy in ways that conventional classroom settings often do not. Direct interaction with natural environments, anatomical models, and geological and paleontological sites helped educators build procedural and inquiry-based skills while encouraging interdisciplinary thinking. 
 

Field experiences also promoted reflective observation, prompting teachers to consider broader social, cultural, and environmental dimensions and to find meaningful ways to integrate scientific content into their teaching. MSSE’s programs nurtured a spirit of lifelong learning, curiosity, and professional growth. Participants reported a renewed passion for science, expanded ecological and cultural awareness, and a dedication to incorporating authentic scientific challenges into their classrooms. 

Overall, the study suggests that thoughtfully designed field experiences empower science educators with the tools, mindset, and enthusiasm needed to create learning environments that reflect NGSS principles—centered on inquiry, collaboration, and real-world relevance. These results will be used in future program assessment reports to measure program learning outcomes (Link to Bridging Field Experiences and Classroom Practice: How Immersive Graduate-Level Science Courses Shape Teacher Self-Efficacy and Experiential Learning Implementation study) 

Submit report to programassessment@montana.edu  

Update Department program assessment report website. 

Update PLO language in CIM if needed (Map PLOs to Course LOs) 

Appendix A 

Student Artifact Examples 

Lesson Plan Example 

Energy Storage Mode Comparison- 

Target Audience: Regular/Honors Physics: Grades 10-12  

Unit: Energy-Energy Storage  

Description: In this lesson, students will learn about the pros and cons of three different energy storage mechanisms: gasoline, hydrogen, and batteries. Students will use their understanding of these storage mechanisms to make a claim regarding what will be the dominant fuel source for transportation in 50 years. They will support their claim using both scientific and societal reasonings.  

NGSS Standards:   

HS-ETS1-1: Analyze a major global challenge to specify qualitative and quantitative criteria and constraints for solutions that account for societal needs and wants.   

HS-ETS1-3: Evaluate a solution to a complex real-world problem based on prioritize criteria and trade-offs that account for a range of constraints.  

Learning Objectives:  

1. Students will be able to explain how hydrogen fuel cells, batteries, and gasoline energy convert stored chemical energy into motion 
2. Student will compare the energy density, efficiency, refueling time, infrastructure, and environmental impact of each fuel 
3. Students will interpret data to compare fuel types 
4. Students will develop a reasonable argument for which fuel could be the most viable in the future  

Materials:  

• Writing Utensil   

• Notecards (1 per student)  

• Laptop (1 per student)  

• Videos and Readings for each fuel type  

Procedure:  

Introduction/Warm-Up:  

1. Open class with the question “What fuel source will use to power our vehicles in fifty years? Why?” 
2. Have students respond to the prompt using a notecard to get students initial thoughts 
3. Gather the notecards and tally student responses on the board 
4. Open the floor for a brief discussion regarding the students’ selections   

The Activity:  

1. Discuss the three currently viable fuel sources: Gasoline, Batteries, and Hydrogen Fuel Cells 
2. Explain that to fully answer the question from the beginning of class the students must educate themselves about the benefits and drawbacks of each of these energy storage modes 
3. Ask students to take out their Chromebooks and go to Google Classroom. Here they will find a variety of resources to use to educate themselves about each fuel type  

Possible Resources:  

https://www.youtube.com/watch?v=UUrmiu5hcyo&ab_channel=CSIRO  

https://www.youtube.com/watch?v=bFb5TKVPSiY&t=4s&ab_channel=EnginuityExplained  

https://www.youtube.com/watch?v=Hatav_Rdnno&t=3s&ab_channel=EngineeringExplained  

https://www.intelligent-energy.com/news/pem-fuel-cells-vs-batteries-for-electric-vehicles/  

https://www.aps.org/archives/publications/apsnews/201208/backpage.cfm  

4. While educating themselves using available resources students should complete the following table to gather evidence and organize their evidence  

5. Finally, students should complete a CER to answer the initial question. For full credit students must address at least one scientific and one societal reason for their claim. 

6. Collect the CERs by the end of class.  

7. Extension: Depending on time a discussion/debate could be conducted  

Key:  

Example CER:  

Claim: I believe that in fifty years most vehicles will be powered by Hydrogen fuel cells.  

Evidence: Reasonable refueling times, high energy mass density, and lack of CO2  emissions.  

Reason: For transportation, it’s important that refueling is convenient and inexpensive while addressing climate change. Hydrogen fuel does this by having similar refueling times to gasoline while having no CO2 emissions (assuming renewable energy creates the hydrogen). Although the infrastructure is limited, I believe that over the next fifty years it is possible to develop this like how we have gas stations.  

Final Exam Example 

Discussion Post Examples 

Final Project Virtual Poster Example 

Appendix B 

Artifacts from Instructors 

Lecture Notes, Prompt and Assessment Example 

Artifacts from BIOE 597 for the program assessment, including examples from lectures, hands-on activities, and assessment  

Prompt: Identify that topics in [chemistry/climate sci/Earth/elementary/life/physics/STEM] are linked together and demonstrate how [chemistry/climate sci/Earth/elementary/life/physics/STEM] is connected to other science content.   

Assessment: A portion of their verbal report asks students to identify a mayfly and describe where it lives, which requires students to connect physics/earth sciences with life sciences to know the correct location within a stream to look for a mayfly and then how to identify that taxonomic group once they find the right location to look.  

Unit Assignment Example 

Rubric Example 

Case Study Rubric