A/B testing in mass messaging

(% class="lead" %)

The System supports A/B testing for each type of mass mailing, allowing multiple message variants to be created and tested to identify the most effective version. For example, it is possible to evaluate how different messages affect the average receipt amount or the average number of receipts per customer.

3

(% class="box" %)

(((

**Contents:**

)))

By default, the same message version is delivered to all recipients in a [[mass mailing>>doc:Main.Smart_Communications.SMC_Use.Mass_messaging.WebHome]]. The **A/B testing **feature allows to send from 2 to 5 different message versions** **as part of a single mass messaging.

12

13

The following items can be changed in different variants of the message:

* Message text;

* Sender;

* Subject (for [[email messages>>doc:Main.Smart_Communications.SMC_Use.Mass_messaging.Email_mass_messaging.WebHome]]);

18

* Files to be attached (for email messages).

(% class="box" %)

(((

**Example of А/В testing **

23

You can create 2 message versions in SMS mailing:

24

25

* Half-price for eco products.

26

* Enjoy a 50% discount on eco-friendly products — just for you.

27

)))

28

29

=== Creating an A/B test in mass messages ===

30

31

To add a new variant of the mailing message, open the page for creating a mass message and go to the **A/B test **block.

32

33

Click **Add variant**.

34

35

|(% style="border-color:#ffffff; text-align:center" %){{lightbox image="A_B_test.png" width="1200"/}}

36

37

You can set the percentage of recipients for each message version. If not configured manually, versions are assigned to customers evenly by default. For example, in case if there are two message versions, 50% of customers receive the first version, and the remaining 50% — the second.

38

39

The percentage ratio can be adjusted using the slider.

40

41

|(% style="border-color:#ffffff; text-align:center" %){{lightbox image="Test_slider.png" width="1200"/}}

42

43

**Example of uneven distribution of message versions:**

44

45

* Version А will be received by 15% of customers;

46

* Version B — 25%;

47

* Version С — 35%;

48

* Version D — 25% of customers.

49

50

|(% style="border-color:#ffffff; text-align:center" %){{lightbox image="Uneven_distribution.png" width="1200"/}}

51

52

When creating an A/B test in a mass mailing, tabs appear in the **Mailing **block, as shown on the screenshot above. The number of tabs corresponds to the number of test variants. Clicking on a tab opens the section for filling in message parameters for that specific variant. There is a need to complete these parameters for each variant.

53

54

To **delete** a test variant, click the recycle bin icon** **in the corresponding tab.

55

56

After completing all message variants, it is possible to send a **test message **using the desired variant. To do this, select the tab with the required message variant and click **Test**.

57

58

|(% style="border-color:#ffffff; text-align:center" %)[[image:Test_button.png]]

59

60

To **preview an email**, select the tab with the required message variant and click the **preview** button on the toolbar.

61

62

|(% style="border-color:#ffffff; text-align:center" %)[[image:Preview.png]]

=== Reports ===

The A/B test report is available on the page where all the other [[mailings reports>>doc:Main.Smart_Communications.SMC_Use.Mass_messaging.Mailing_report.WebHome]] are located.

67

68

The first table shows the following information on each group:‌

69

70

* **Number of recipients**: number of messages sent to this group;

71

* **Message delivered**: number of messages delivered in this group;

72

* **Number of customers who made a purchase**: calculated based on [[attribution parameters>>doc:Main.Smart_Communications.SMC_Use.Receipt_attribution.WebHome]];

73

* **Conversion to customers**: percentage of customers who received a message and made a purchase due to the mailing;

74

* **Number of receipts in a group**;

75

* **Average number of receipts per customer**;

76

* **Amount of purchases**;

77

* **Average revenue per customer**.

78

79

The second table contains a detailed information about **conversion** and about **unwanted behavior **in each group. For example, in group B:‌

80

81

* **Sent** — 36,057 messages (100%);

82

* **Delivered **— 36,057 messages (100% of messages sent);

83

* **Opened **— 918 messages (2.5% of messages sent);

84

* **Clicks** in 198 messages (0.5% of messages sent);

85

* **Orders** — from 26 messages (0.1% of messages sent);

86

* **Unsubscribed **after receiving a message — 2 persons (0%);

87

* **Spam complaints** – 0 (0%).

88

89

|(% style="border-color:white; width:1083px" %)(((

90

The values in some columns are **highlighted in colour**: green - the best value, red - the worst.

91

92

For example, Group A has an average revenue per customer of **25 USD**, which is the best value among all the groups (highlighted in green). In group B - **10 USD**, this is the worst value (highlighted in red).

93

94

(% class="box infomessage" %)

95

(((

96

**Note**: the report shows attributed receipts associated with the mailing. More information about attribution can be found in the [[Receipt attribution >>doc:Main.Smart_Communications.SMC_Use.Receipt_attribution.WebHome]]article.

97

)))

98

)))

Navigation

Wiki source code of A/B testing in mass messaging