All You Need to Know About Statistics in <300 Words

All You Need to Know About Statistics in <300 Words

And why you should never use them!

June 20. 2023

Maryfrances Porter

---

I got a great comment on a video on my YouTube channel saying that even though I tell nonprofits that they should NOT be doing program evaluation, nonprofit leaders still need to be able to engage in statistical thinking (just like a general practitioner is familiar with the science of statistics).

I definitely believe we should all have the basic ability to be wise consumers of data, because there’s a lot of bad and misleading data out there. But, I think it goes a step too far to say nonprofit leaders should be able to engage in “statistical thinking.” (To all the statisticians out there: I know I’m really simplifying things. Bear with me!)

 

All you need to know about statistics in <300 words

The bare bones essentials about statistics:

1. They are a scientific tool. Their utility depends – in large part – on how the experiment is set up.
2. They are used to test a hypothesis (e.g., the students in the treatment group will have improved school attendance).
3. The simplest statistical test looks at the average outcome for both groups to see whether they are mathematically different. You want to be at least 95% sure the groups have different outcomes.

Being a wise consumer of data means understanding what statistical significance does NOT tell you. . .

1. …what the averages are or how far apart the averages are (i.e., how big the difference is between groups).
2. …whether the averages are far enough apart for it to be important. If you were delivering a treatment to increase school attendance, you might care a lot if there is a 20-day difference between the group averages (e.g., on average, the treatment group attended 20 more days than the other group), but you might not think it’s important if there is only a 2-day difference. Depending on a number of factors, both results could be statistically significant.
3. …how likely people in the treatment group are to have an important amount of improvement (e.g., at least 20 more days present).
4. …what any one person’s experience will be. This is because people are not averages and life is complexity.

If you want a real example of scientific prediction and how it relates to your life, think about weather forecasts. Think about how the forecast for a single day changes over time. Think about how often it the forecast turns out to be wrong. The meteorologist might have been 95% sure that there was a 50% chance of rain today. But it didn’t rain.

(299 words!)

 

Why you should never use statistics

Now let’s say your nonprofit values school attendance as a pathway to high school graduation and adult prosperity. Therefore, you give your student clients the above intervention because there is evidence it works. This is an evidence-based practice.

How many of your clients improve their attendance? How many improve by at least 20 days?

Statistics don’t work for nonprofits because:

1. You are a practitioner, not a scientist. You’re delivering an intervention in a complex and sometimes chaotic environment. You are not running a well-designed experiment.
2. You are not testing hypotheses. You are simply giving people what science says helps.
3. You are looking for results for individuals, not results on average. On an individual basis, only their experience of the outcome (how many more days they attend school) matters in their life. Whatever the average result is doesn’t matter to your client.

 

What does this mean about using evidence-based practices

Social problems are not bacteria – there isn’t an inoculation for any of them.

Said another way, not every student who gets an evidence-based intervention will have meaningfully improved attendance.

Evidence-based practices are just more likely to produce good outcomes, compared to no intervention or other practices.

 

A real life example

The Perry Preschool Program is one of the biggest successes of social science in the 20th century. Children who got the program in the mid-60’s were less likely to have become pregnant as a teen, to have been to jail, or to be on public assistance by the age 40. Preschools across the nation use the Perry Preschool curriculum.

What do the results really say?

• 6% of Perry students had teen pregnancies compared to 12 for non-Perry students. If you had 100 Perry students, 6 would get still get pregnant as teens.
• 28% of Perry students went to jail compared to 52% for non-Perry students.
• 59% of Perry students had been on public assistance within the past 10 years compared to 80% of non-Perry students.

People who got the very best preschool curriculum, delivered by the actual scientists who invented it, still had teen pregnancies, went to jail, and were on public assistance. However, finding any impact 40 years later is very, very, very rare – so these results reflect a remarkably powerful intervention.

Using an evidence-based practices give your clients a better chance of success. It does not inoculate them against human struggles. 

 

Learn more in the ImpactStory Academy!

---

Partnerships for Strategic Impact

At Partnerships for Strategic Impact^®, I have all the tools to build out the structures and strategies you need for collecting actionable impact data; I train staff in how to use them, analyze data, and make action steps for change; I teach you to tell powerful impact stories. My goal is your sustainability: I provide just the right amount of coaching and ongoing support to ensure success.

This is within reach. It’s not harder than managing your budget. Plus, you, your funders, and the people you serve deserve to know more about the value your organization delivers.

If you’re reading this, then you’re already part of my network. Schedule some time on my calendar. There’s no sales pitch and no obligation. Just curiosity and answering your questions. I am the seasoned program-evaluation, program-development, data-storytelling consultants you need, right in your back pocket. I am here to bring order to the process of figuring out how to track and use your impact story.
 

– Maryfrances

Your ImpactStory™ Coach