Last Updated: MAY 3, 2019

Program Evaluation Highlights

NASA's Universe of Learning Program

 
 

NASA’s Universe of Learning engages two external evaluators, Goodman Research Group (GRG) and Cornerstone Evaluation Associates LLC (CEA) to provide rigorous external evaluation of the NASA’s Universe of Learning (UoL) portfolio of products, programs, and professional development offerings, as well as to evaluate the multi-institution partnership itself. GRG serves as the lead evaluator to assess the effectiveness and impact of three program elements: Data Tools and Participatory Experiences, Multimedia and Immersive Experiences, and Exhibits and Community Programs, as well to assess the dynamics and partnership of the NASA’s UoL team. CEA serves as the evaluator for the Professional Development program element.

 
 

Overall Findings: 2018 Examples

  • NASA’s UoL has defined its metrics for success and is implementing them.

  • The NASA’s UoL partnership continues to strengthen and demonstrates effectiveness on key research-tested criteria of successful collaborations.

  • A set of evaluation studies suggests NASA’s UoL is meeting the needs of its audiences and making progress toward SMD objectives/metrics.


Data Tools & Participatory Experiences

NASA’s Astrophoto Challenge

130 recruited participants completed a pre-survey online, were provided a link to the NASA’s Astrophoto Challenge, and then completed a post-survey the next week. Respondents were divided into two groups:

  1. No/Minimal engagement (did not process an image)

  2. Greater engagement (processed image, compared image to NASA image, and/or submitted to Challenge)

Changes in understanding of how astronomical images are collected and created were evaluated by coding responses to a “test” question into four levels:

  1. No understanding

  2. Basic understanding (images are collected using telescopes)

  3. Deeper understanding (detectors on telescopes, digital nature of images)

  4. Sophisticated understanding (processing, digital enhancement, colorization)

Greater engagement in NASA’s Astrophoto Challenge predicted increased understanding of how astronomical images are collected/created.

 

Multimedia & Immersive Experiences

ViewSpace

After watching ViewSpace videos, respondents knew more about the electromagnetic spectrum, the life and death of stars, and “how astronomers know.”

What respondents learned (examples), in their own words …

  • There is a lot out there that the naked eye and even regular telescopes cannot see without using special technology such as infrared and Xray light.

  • That there is much more going on besides darkness and stars … a disk of rock and ice just floating around stars … wow.


Exhibits & Community Programs

Girls STEAM Ahead with NASA (GSAWN)

In 2017 & 2018, GRG collected brief paper surveys from youth who had participated in GSAWN events.

  • In 2017, 116 participants from 5 events completed surveys.

    • 86% girls / 13% boys

  • In 2018, 114 participants from 5 events completed surveys.

    • 54% girls / 46% boys

Among girls at GSAWN events, impact on learning and interest is greatest for those who meet a scientist. This effect is not observed among boys.

 

Professional Learning Experiences

NASA’s Universe of learning Science Briefing

  • In July 2018, all individuals who had attended at least one of the ‘live’ science briefings were contacted via email and invited to participate in the online, comprehensive impact survey.

  • Emails successfully reached 324 science briefings’ attendees.

  • In total, 96 entered the survey for a response rate of 30%.

Highlights

  • Sharing content/resources: 11,370 people were introduced to NASA’s UoL content/resources by 55 respondents who estimated sharing

  • Likelihood of using content/resources: Respondents are highly likely to use NASA Astrophysics content/resources to design learning experiences.

  • Personal impact of science briefings: Respondents reported increased interest in and understanding of NASA Astrophysics, as well as boosted confidence in conveying information to the public.

  • Learning from SMEs: Respondents reported that it was extremely important for them to learn science content directly from scientists. Having direct access to the scientists also enhanced their sense of community.