A/B Test Workflow Tutorial

Use A/B Tests to experiment with different variations, or variants, of a message for a given audience, and provide engagement reporting data for each variant.

Send a push notification, in-app message, or a combination of the two. Learn more in Message and Workflow Types and the Workflow Overview.

What You'll Do

In this tutorial, you will:

  1. Define who will receive your message variants.
  2. Name your test, enter the number of variants, and adjust the audience vs. control group sizes.
  3. For each variant, choose which message type to send and enter its content.
  4. Direct the users' next step after they tap or swipe the message variants.
  5. Decide how and when you want the message variants delivered.
  6. Preview and send the message variants.

Features and options are explained along the way.

Tip

You can configure the default appearance of your In-App Messages in Settings: Configuration: In-App Message Design. This includes button text and message color, screen position, and more.

Steps

Open your project from the dashboard, then click Create and select the A/B Test workflow.

Audience

Note

The audience you select in this step defines the entire set of users for the A/B test, which will include the control group (users who receive no message) and users who will receive different messages, according to the variants that you define.

  1. Optionally modify which Platforms are included in your audience.All platforms configured for your project that are available for A/B testing will be listed and enabled by default, but you can toggle to limit your audience to specific platforms. You must select at least one.

  2. Select the radio button for your desired recipient group.

    • All Users sends the message to your entire audience.

    • Target Specific Users lets you create a recipient group based on segmentation data. See: Target Specific Users.

    • Test Users are predefined recipient groups. After choosing Test Users, select from the Test Groups dropdown menu that appears. See: Audience: Test Groups.
  3. Click Variants in the header to move on.

Variants

  1. Enter a descriptive Test Name.
  2. Select the Number of Variants for your test, up to 26. The default is two. You will be able to add or delete variants in later steps.

  3. Use the slider to adjust the percentages of your target audience vs. control group. By default we send your test to 80% of your audience, keeping a control group of 20%

Content

Variant A Set Up

  1. Enter a name for the first Variant.

  2. Choose between sending a Push Notification or an In-App Message, or choose Silent Push Notification to send a push without notification text.After you make your selection you have the option to combine message types, then click Continue to confirm your choices.

  3. Enter the text that will display in your message. A preview will display as you type.
    Note

    If you chose to combine a Push Notification and In-App Message, an In-App Message Text section appears below the message truncation indicator. By default, the In-App Message's alert text is the same as the Push Notification text. If you'd prefer to use a different message, select the radio button for Write alternative, and enter your text in the box that appears below.


  4. Select a Message Action from the dropdown menu.
    Note

    The Message Actions listed in the dropdown menu differ depending on which message type you chose and if you chose to combine message types. See: Message Reference: Actions.

  5. Optionally Set a tag.

  6. Set Optional Features.

  7. Configure additional variants. Select the next variant by using the lettered tabs, and complete the name and message setup for each.

    For variants B-Z, you have the option to either use the content from a previously created variant, or start from a blank message.

    You may also add and delete variants using the tab X and + options.

  8. Click Delivery in the header to move on.

Delivery

  1. Click to select Send Now or Schedule.
    • Schedule: Enter a date in YYYY-MM-DD format and make your selections from the dropdown menus.
  2. Set Optional Delivery Features.

  3. Click Review in the header to move on.

Review

  1. Select a variant from the dropdown menu in the Content section or above the preview.

  2. Review the device preview and message summary. Click the arrows to page through the various previews. The platform and display type dynamically update in the dropdown menu above. You can also select a preview directly from the dropdown menu.
    If you would like to make changes, click the associated step in the progress header, make your changes, then return to Review.

  3. Send your message! Click Send Message or Schedule Message at the bottom of the window.
Tutorials