Mindvalley

Mindvalley Boosts Ad Performance by Managing User Engagement: An A/B Test

Actively moderating and engaging with comments leads to increased brand trust, customer engagement, and ultimately a higher return on ad spend.

+54% Conversion Rate*

+48% ROAS*

+47% Purchases*

*in the Treatment Campaign, compared to the Control Campaign

A/B test approach to compare the impact of comment management with two campaigns that had identical targeting, creative, setup, and budget. Both campaigns ran at the same time from April 4, 2023 - May 2, 2023.

OVERVIEW

Exploring the impact of comment management on performance metrics

Social media advertising is a constantly evolving space, with a multitude of factors influencing the success of your campaigns. It can be a daunting task to navigate through the different variables and determine the most effective strategies to ensure your campaign's triumph.

This study delves into the connection between managing comments on social media (hiding harmful comments and responding to customers) and performance metrics. By conducting an A/B test on two campaigns, this research examines the effects of comment management on campaign outcomes.

The results of the study indicate that managing comments has a positive impact on social media campaign performance in several areas, including ROAS, number of purchases, cost per result, and others. By emphasizing the significance of actively managing user comments on ads, this study offers valuable insights for social media advertisers looking to optimize their campaigns.

THEIR STORY

About The Brand

Mindvalley is an online personal growth and education platform that offers various courses, programs, and content to help individuals enhance various aspects of their lives. It was founded in 2003 and has grown into a global community of millions of learners.

Mindvalley's primary focus is on providing transformational learning experiences in areas such as personal development, health and fitness, spirituality, entrepreneurship, and more. They collaborate with renowned experts and teachers worldwide to curate and deliver high-quality educational content through their online platform.

Methodology

• Two campaigns sets were created by Mindvalley, representing the treatment group (campaign A) and the control group (campaign B).
• The campaigns were exactly the same in terms of targeting, creative, setup, and budget.

Both campaigns ran at the same time from April 4, 2023 - May 2, 2023.

• The total budget for this A/B test was $20,000, and each campaign spent approximately $10,000.
• The treatment group had comments managed by BrandBastion, a fully managed comment management solution that responds on behalf of the brand to specific categories of comments and hides harmful content.
• The control group did not have any comment moderation or responding in place, allowing comments to remain unmanaged.

BrandBastion Coverage on the Treatment Campaign

Solutions in place

BRAND PROTECTION [FULLY MANAGED]

• BrandBastion monitors all comments in real-time to hide harmful content such as spam, scam, hate speech, brand attacks and critiques, violent or inappropriate comments.
• BrandBastion sends alerts to the brand when immediate attention is required.

COMMUNITY MANAGEMENT [FULLY MANAGED]

• BrandBastion responds on behalf of Mindvalley to Fan Community, Purchase Intent, FAQs, Constructive Feedback and Complaints 24/7.

Tyler Foo

Head of Advertising - Mindvalley

"We recently teamed up with BrandBastion to run an A/B test. We wanted to see if managing comments would impact our performance. So, in one of our campaigns, BrandBastion took charge of handling all the comments, while in the other campaign, we simply let the comment section go unmanaged. And you know what? The campaign where BrandBastion managed the comments outperformed the other one by a long shot. We saw some serious improvements in our metrics like higher return on ad spend (ROAS), more purchases, better conversion rates, and a decrease in the cost per result."

Managing comments has significant positive impacts on ad performance

RESULTS

By actively moderating and engaging with comments, brands can create a more positive and trustworthy environment, leading to increased brand trust, customer engagement, and ultimately a higher return on ad spend.

Based on the sample available in this study, there is evidence that campaigns incorporating comment management will result in:

• an increase in ROAS, at 87% confidence*
• an increase in purchases, at 88% confidence*
• an increase in Conversion Rate, at 89% confidence*
• a decrease in cost per result at 87% confidence*

These findings underscore the importance of comment management as a strategic approach to optimize ad performance and maximize the return on investment in social media advertising campaigns.

*Confidence can be understood as the probability that the observed results is not caused by chance or natural variation. It was calculated by using independent samples t-test and two proportion z-tests (where appropriate) with Null Hypothesis that the metrics in question were not improved by the treatment.