Find information about general impact measurement practices and how to put Impact Navigator’s Indicators into place.
Whether you’re a research novice or a pro, let’s dig into the technical details of applying Impact Navigator’s methodology to your work.
Background
Impact Navigator draws from the social science research framework called Positive Youth Development (PYD). Using a unique combination of PYD and new crowdsourced information from Headstream’s research project Digital Delta, Impact Navigator has curated the most applicable and impactful tools to observe, measure, and document how your innovation affects young people.
What to measure
Impact Navigator’s impact indicators (Indicators) are a series of self-reported survey questionnaires that measure 15 critical areas of wellbeing in PYD:
- Bonding
- Social Competence
- Emotional Competence
- Cognitive Competence
- Behavioral Competence
- Moral Competence
- Self-Efficacy
- Prosocial Norms
- Resilience
- Self-Determination
- Spirituality
- Future Beliefs
- Identity Development
- Prosocial Opportunities
- Prosocial Rewards
Download the complete framework here.
Finding the impact goals and relevant Indicators that match your mission is the first step to advancing your measurement journey.
Now let’s dig into the details of applying Impact Navigator’s methodology to your work.
How to measure
Each Indicator’s scale can be incorporated into the system you are currently using to:
- Administer surveys
- Track responses
- Observe engagement data of users’ interaction with your product
You can ask young people to respond to each scale item (or survey question) based on the Likert Scale provided. The scores from these scales can be calculated in a few ways:
- Most scores on the scale can be calculated by adding the scores on each item to create a total score
- Some scales will require that you reverse code specific items that are written in the negative, and then add the rating for each item to obtain a total scale score
- Some scale scores are based on an average score
We generally recommend administering all scale items in a bundle to maintain reliability and validity.
When to measure
By asking users to complete the Indicator scale questions at repeated intervals you can measure the impact of your product on targeted outcomes over time. We recommend measuring at the following points in time:
- When the user starts to use your product (baseline/pre-test)
- Then at least every four to six weeks
- When the user stops using your product (post-test)
Why measure in this way?
Remember that impact measurement is not only to find “good” news or data that validates your original hypothesis. Impact measurement can help you determine ways you can reinforce and develop your product, and can serve as a feedback mechanism regarding how your product contributes to any changes (positive, neutral, negative).
If your users demonstrate a positive change in their scores on the indicator scales, you can:
- State more confidently that your offering promotes that outcome for young people
- Use these data to document that your product led to measurable, positive changes based on empirically-based measures
- Note what other things might have contributed to this positive outcome. For example, a positive role model might have come into their life. As such, be careful about how you interpret these data and take credit for changes
If your users demonstrate a neutral or negative trend in their answers, you can:
- Use these insights to make targeted improvements to your product (compared to redesigning the entire product)
- Drive alternate value propositions to your users
- Adjust your social impact strategy
There are numerous benefits of using an evidence-based outcome measure, like:
- Confidence that the measure has been subjected to testing to establish its psychometric properties
- Confidence that the measure is valid (meaning it measures the concept that it’s matched with)
- Confidence that the measure is reliable (it consistently measures the same concept, which minimizes measurement error)
- The measure has been standardized, which allows for comparisons across settings and products
- Confidence and credibility that there is a relationship between your product and observed changes in the Indicator for young people
Risks
Impact Navigator is limited in what it can offer you, especially during beta testing, so it presents a few risks that we want to support you with. Please reach out to us to discuss these risks in depth!
- Attribution – your users do not live in a vacuum, so overly claiming full credit for changes they might experience could lead to what researchers call “attribution risk”
- External Validity – this refers to the ability to generalize based on research findings. While you may observe changes in your study groups, be careful about extrapolating insights for populations that might not share the same attributes and environments
- Control Groups, Comparison, and Benchmarking – It can be difficult to source and study control groups to compare your users against. Impact Navigator always promotes using a control group to compare and benchmark your findings against
- Present Bias – It is tempting to take “point in time” counts, and stop measuring after gaining a snapshot of users in the present moment. It’s important to offset present bias by maintaining research practices and analyzing trends in data over time
- Confirmation Bias – We are hardwired to look for good news and to confirm our ideas. Look out for places where you might design your evaluation practices with confirmation bias
————————————————
As with all of our materials, we welcome your feedback and collaboration on improving future iterations of this guide! We hope you found this useful.
About Impact Navigator’s Positive Youth Development Methodology (FAQs)
Where did these measures come from?
The measures, or instruments that are comprised of scale items (survey questions), came from research studies within positive youth development (PYD) research. Impact Navigator selected PYD because it:
- Is prosocial
- Engages youth within their communities, schools, organizations, peer groups, and families in a manner that is productive and constructive
- Recognizes, utilizes, and enhances young people’s strengths
- Promotes positive outcomes for young people by:
- Providing opportunities
- Fostering positive relationships
- Furnishing the support needed to build on their leadership strengths
How are these different from psychological and/or clinical health measures like the PHQ-9 for depression?
PYD constructs and their measures are rooted in social science and are often used by social workers and social work researchers. Compared to psychological or clinical health measures, these measures aim to comprehensively understand adolescents’ experiences and the ways research has connected that to their overall healthy development. Impact Navigator views these PYD constructs as “upstream” effects that are known to have positive implications for mental health and wellbeing. As such, they don’t use biological health and psychological measures to assess changes in young people’s diagnoses or health events, which require academic and ethical safeguards that are best met in a clinical research setting rather than the DIY entrepreneurial environment most Impact Navigator users operate in.
How does this help me understand the effects of my work on users’ mental health?
PYD is a strengths-based model designed to consider adolescents’ developmental plasticity, internal developmental assets like psychological competence, and external developmental assets like community influence. As such, Impact Navigator sees that measuring PYD effects could position innovators well to draw strong correlations to improvements in overall wellbeing. Impact Navigator is currently assessing ways to build in additional mental health constructs to its operating framework in future expansions.
We are excited for you to help us test this new tool.
Thank you so much for testing this new tool with us during Impact Navigator’s pilot phase.
We built this because we know youth wellbeing startups care deeply about creating more positive, safe experiences for young people. We work every day with entrepreneurs building education, health, and social technologies to make the internet a more beautiful place to grow up. But how can we take the next step beyond that commitment, towards implementable practices that tell us if our innovations are having the effects we desire?
That’s where Impact Navigator comes in. This novel framework will help guide you from your value propositions to your adolescent development outcome goals in ways you can observe, track, measure, and ultimately use as meaningful performance insights.
We know this can be complicated work. So wherever you are on your impact measurement journey, you might have some questions about how to use this device, so we’ve provided more information in the How-To Guide.
As always, reach out to us with your additional questions through the site’s feedback button or directly to impactnav@headstreaminnovation.com. We invite your ideas, revisions, requests, and partnership to help build the next iteration of Impact Navigator!
