Multivariate & A/B Testing
Automatically test your notifications
API
The A/B Test is a technique used by marketing and communication teams to optimize the content of a campaign. It consists of testing different versions of a campaign to see which one is the best.
The indigitall platform offers a 'intelligent' version based on Machine Learning of this functionality so that this process is very simple and comfortable, so that you can use this A / B test technique regularly in your day to day day, and in this way you can optimize your campaigns based on real data of user response.
Step 1. Create an A/B Test
The creation of each test is easy. On the menu, choose the Campaigns/Push A/B Test tab.
Then you can view all created test campaigns and moreover clicking on each button A or button B, you will see all results of each test.
To create a new test, you will click the green button at the top right area. Then you only need to write a name for the campaign.
Campaigns
Choose the both campaigns that you have created previously in the AppPush/WebPush section.
Filters
In this area, you have some options to select your audience by topics, external applications, geographical filters, frequency of use, device types or simply updating the database. Please check all information about filters here.
Remember that you need to upload your contacts with a .CSV file with all deviceIds and the variables you have used on the campaigns.
Configuration
In the configuration area, you choose the day and time to start and finish the campaign.
Finish
Click on the Save and Send button of any tab.
How Intelligent Send Works
The method we use is known as Multi Armed Bandit testing, MBA, inspired by the algorithm of Thompson Sampling.
While A/B tests with manual send are based on exploration, collecting data to see which is the best version and making a decision, MBAs are designed for exploitation, showing the version that seems best while testing continues.
Our algorithm progressively and automatically sends one of the versions to all devices. During the first hour we will make four shipments with all the versions and thus be able to analyze which one seems better. From the second hour every fifteen minutes and depending on the success of the versions, the one with the highest conversion rate will be sent to more users.
The longer the test lasts, the better the math model will learn and send the most successful version to more devices. Notifications will be sent in a time interval of 1 to 7 days in a constant and gradual way in daytime, depending on the number of total devices and hourly filters.
A/B Test Statistics with Intelligent Send
By clicking on the statistics button in the section Campaign> Push Test A/B, you will be able to check the notifications sent and how many have been clicked by each version and device : web, android, iOS as the total.
You will get a view like the following:
On the left is the detail by campaign and in the last line the total of the campaign:
- Received: Number of devices that have received it. It can be less than those sent if there has been an error in the reception of a sent push.
- % Received: Percentage of received with respect to the total received of the campaign.
* Clicks: Number of devices that have clicked. - % Clicks: Percentage of clicked with respect to the total received.
On the right is the total for the campaign:
- Devices: Total number of devices
- Sent: Total and percentage of devices sent out of the total devices in the campaign.
- Clicks: Total and percentage of devices that have clicked on the total number of devices in the campaign.
When clicking on a version, a pop-up window appears with the details for each device (web, android, iOS) and the total CTR (Click Through Rate), the number of notifications clicked among those received.
Status of an A/B Test
Created tests can be in one of these states:
- Completed: Has been sent to all devices in the campaign.
- Sent: Manual send of all versions has been completed.
- In progress: The campaign is being sent by intelligent sending.
- Canceled: The test has been canceled.
- Error: An error has occurred.
- Pending: The test has been created successfully but the intelligent send has not started yet.
Updated 7 months ago