Scientific Rigor and the Social Sector
August 14, 2024
Maryfrances Porter
The Highlight of My Last Week!
The I had the unexpected and delightful opportunity to have a conversation with David Fetterman last week!
Dr. Fetterman is a renowned program evaluator who created Empowerment Evaluation. He’s one of the earliest advocates of empowering communities to ask their own questions and come to their own conclusions. Basically, one of my heroes!
As he and I were talking, I was thinking about the differences between what we mean by scientific rigor and what it means to collect credible and rigorous data in the social sector.
Different Strokes for Different Folx
You’ve heard me say it before: Program evaluation is NOT your job!
Nonprofits, and their funders, are partnering as practitioners to support individual thriving in communities. While some social sector organizations are in fact focused on doing science, the vast majority of you are doing the hard, hands-on work out in the real world.
Where does your work fall in the flow of how we serve people with high quality programs? Well, there are three steps to getting an evidence-based program into a community:
STEP 1: Research into how people and society work: This (often academic) research focuses on discovering the root causes of human thriving and challenges. This is basic science and it tests theories. THIS IS NOT YOU.
STEP 2: Development of programs and program evaluation: This is all the work behind how a program becomes an “evidence-based program.” This is applied science (often academic, but can be done by private or public organizations), where theory is adapted into a program which is tested out with real people. Ideally, a program is evaluated more than once. THIS IS NOT YOU.
STEP 3: Actually providing an evidence-based program to people! THIS IS YOU!
Defining Scientific Rigor
The first two steps – basic and applied science – are science. Science is often funded by the federal government, like the National Institutes of Health or the National Science Foundation. Some foundations, like the Gates Foundation and the WT Grant Foundation also fund scientific research.
Basic and applied research require scientific rigor, which includes:
- Scientific design, including robust sample size and appropriate statistical analysis
- Objectivity and transparency
- Institutional Review Board (IRB) scientific ethics review
- Peer review and publication
- Ideally replication (the research is done a second time)
Defining Rigor in Practice
As a practitioner – out in the world doing step three – rigor is also important, but it looks very different. I’ve seen nonprofits and funders asked to adapt scientific rigor, but you and I both know that doesn’t work out very well!
Good news…Bad news: You do not – and cannot – be scientifically rigorous with your data. But, you do need to thoughtfully and intentionally engage in rigorous practices. Let’s call these high quality practices.
Rigorous, High Quality Practices
Let’s think about a doctor or teacher. They are both using evidence-based programs (e.g., medicine or teaching techniques), which have already been shown (by basic or applied science) to work.
(1) You are educating yourself about patient/student strengths and needs, as well as the medicine/interventions most likely to help.
It is rigorous when you rely on high-quality information.
(2) Since there aren’t enough medicines and evidence-based programs to address every challenge for every person in every situation, you adapt your practice to meet the specific needs of the people you are supporting.
Doctor: I think this calls for using a medication off label.
Teacher: Hmmm, this student isn’t learning as expected. Let me find out more and try something different.
It is rigorous when you are responsive, thoughtful, and responsible about adapting what you do.
(3) You understand what people take away from working with you (e.g., scheduling a follow-up, checking in, or testing knowledge gained).
It is rigorous when you intentionally and consistently track the progress patients and students make, as well as ask about what they received, the relevance of what they received, and their satisfaction with the experience.
(4) You are looking at the data and feedback, taking it in, contextualizing it, checking your assumptions about what it means, and making true statements about what you’ve figured out.
Doctor: Patient A got better without medication, maybe they didn’t have the diagnosis I originally thought.
Teacher: Student B really struggled with that concept last week. I wonder if I wasn’t explaining it well or if they were distracted by something?
Fun fact: You do not need to make public all the disappointing data. You do need to incorporate the disappointing data into making improvements in your practice. But – be careful about making positive public statements which include lies of omission.
This is rigorous when your conclusions are inclusive, contextualized, acted upon, and you are making true public statements.
Partnerships for Strategic Impact®
At Partnerships for Strategic Impact®, I have all the tools to build out the structures and strategies you need for tracking and using impact. I train social sector staff in how to use these tools, in how to analyze data, in how to make action steps for change, and how to tell powerful impact stories. My goal is your sustainability: I provide just the right amount of coaching and ongoing support to ensure success.
This is within reach. It’s not harder than managing your budget. Plus, you, your funders, and the people you serve deserve to know more about the value your organization delivers.
If you’re reading this, then you’re already part of my network. Schedule some time on my calendar. There’s no sales pitch and no obligation. Just curiosity and answering your questions. I am the seasoned program-evaluation, program-development, data-storytelling partner you need, right in your back pocket. I am here to bring order to the process of figuring out how to track and use your impact story.
– Maryfrances
Your ImpactStory™ Coach
Want even more?
Check out my Blogs and Videos
Check out the Resource Library
Follow me on all the places #ImpactStoryStrategy and @impactstorystrategy
