How we increased the results from P2P fundraising with +36% through machine learning

Since we launched the Fundraiser Coach - our machine learning algorithm -the average fundraised has risen by 36%. Here is how we found that result.

Updated with latest data: January 19th

The impact of our machine learning model

We found that the average fundraised for P2P fundraisers increased by 36%. The average fundraised went per fundraiser went from 434€ to 636€ over 12 months. This period coincides with the 12 months following the launch of a new way to recommend actions to P2P fundraisers, based on our own machine-learning model.

When we wrote the grant application from Innovation Fund Denmark, we estimated that we could get a 25% increase. We found a 36% increase. A 36% increase is out of his world! It is above all expectations, and we frankly can’t get our hands down.

‍

What does the fundraising  coach do? 

We call our machine learning model “The fundraiser coach” because it coaches P2P fundraisers. The goal is to coach the fundraisers into being as good as they can. 

When a supporter creates a fundraiser, we calculate the predicted amount it will fundraise, given our initial knowledge. Afterwards, the model calculates the impact of specific actions we can recommend them to do. 

A recommendation could be to give the first donation yourself, write a more extended description or add an image. The action with the highest calculated impact is what we recommend the supporter to do. This process is then repeated multiple times throughout the lifetime of the fundraiser. The calculations are done by a model trained on the data from all previous fundraisers created through BetterNow.

The recommendation is then served through a dashboard for the fundraiser called ‘fundraising tools’ and by email. In addition, we added a few other changes to encourage self-donations and a few other changes. Thus the 40% can solely be traced back to the machine learning model, but also the many minor improvements that were born out of that process.

We have more details on the fundraiser coach here, how we developed it, and how we have thought about the data ethics surrounding it.

Our analysis

We have added the following section because we want to be as transparent as possible with this result. When an effect is as massive as +36%, we expect you to question it. We want to be open on why we calculated the result as we did and included some caveats on the interpretation of it.  

Cohort analysis

We do cohort analysis of fundraisers to assess the impact of new features. A cohort analysis breaks the data into groups of data before the analysis. In our case, we break it down by calendar time. So we want to see how fundraisers created in a specific time frame perform compared to fundraisers created in previous periods. Breaking it down by month is the best way for us, as the data gets too noisy and too prone to outliers if we make it in more granular time frames (such as weeks or days).

This analysis started by eyeballing a chart in our BI (business intelligence) tool and seeing that the average raised per fundraiser seemed to rise over the last 12 months. But even at monthly groupings, the data is noisy. Seasonal effects have a significant impact, and even at our scale, a single fundraiser can move the average for a month.

A 12-month moving average

A common strategy for dealing with noisy data is to make a moving average. A moving average will constantly update itself by changing the period over which it is calculated. 

In our case, we have used a 12-month moving average. A 12 -month moving average uses the preceding 12 months to calculate the average for a month. So the average calculated for July 2022 will be calculated over the period from June 2021 to July  2022. This way, we smooth out all seasonal effects and see the underlying trend in the data.

It is problematic to take an average of averages. Thus we have weighted our 12-month-moving average by the number of fundraisers in each month.

Adjusting for outliers

P2P fundraising has extremely skewed results. In some cases, the top 1% of fundraisers will constitute 50% or more of the results for a charity. The last 12 months have had one big event that made us question the results: the Ukraine war. We can easily see that the fundraisers in march had higher averages than other months, so there is reason to suspect that this drives part of the result. We have removed outliers to help remove the effect of any specific campaign and fundraisers.

But when is a fundraiser an outlier? We usually treat any fundraiser with more than 5.000€ as an outlier. So in our base scenario, we exclude all fundraisers that have fundraised more than 5.000€. 

That gave us a new question? Maybe 5.000 € was too low or too large? So to answer that, we did a sensitivity analysis of various levels from 1.000€ to 10.000€. 

Adjusting the outlier level moved the entire graph up and down rather than changing its curve. So if we picked a low level, the average increased over all months. And if we picked a high level for outliers, the average fundraised increased uniformly. Thus the outlier level did not have a significant impact on the result.

Therefore we decided to continue with our base scenario. Here an outlier is a fundraiser above 5.000€. 

As a result, the actual numbers underestimate the actual performance, as we have omitted all the top fundraisers from the analysis.

The final graph and data

The graph below shows the average fundraised rising early in the life of BetterNow. Then it flattens out, and not much happens. Then in July 2021, we see a steep increase. 

Interpretation

We have, over the years, kept adding more features and ways of fundraising. Between 2014 and 2020, our focus has primarily been on payments, administrative functions and integrations. So it is no surprise that we haven’t moved the average fundraised during this period. It also makes sense that it increases in the early life of BetterNow. This period of rapid iterations on design, UX and email guidance to fundraisers.

Then in July 2021, we released our most prominent new feature in a long time. This feature was the Fundraiser coach. As described above, it was not just a way of generating recommendations for fundraisers based on a machine learning algorithm but a long list of minor improvements based on our data analysis. 

As the graph is a 12-month moving average, this will only have a slight impact initially, and after 12 months, the whole effect is present. And we do see this, the following 12 months of this release, we see this steep increase in average fundraised. 

But is it a temporary effect? Well, we won’t know until later. So we will promise to revisit the analysis later when more data has been added. As fundraisers often last for two or more months, we can’t use data from the last 2-3 months as the fundraisers haven’t finished fundraising. We hope to make an update around February next year when we have another six months of data.

What about inflation?

The last year has seen relatively high inflation. Could this explain the increased fundraised amounts? A reasonable measure to use in adjusting for this would be core inflation which has been around 6.5% in the countries we operate in over the same period. However, it is questionable if Inflation would positively impact donation amounts and, even if, it is not, what is driving the results in any way.

Conclusion

We are confident that the major feature releases in the summer of 2021 drove the result. Even though the increase started way before the march 2021 invasion of Ukraine, we can still see this period having a positive impact on the total effect, and we can’t rule out that some parts of the increase only came due to this.

With a 36% increase, there is plenty of room for still having a massive positive result, even though some of it comes from external sources.