A/B Tests Composer Tutorial

Use A/B Tests to experiment with different variations, or variants, of a message for a given audience, and provide engagement reporting data for each variant. Learn more in the Composer Overview.

What You’ll Do

In this tutorial, you will:

  1. Define who will receive your message variants.
  2. Name your test, enter the number of variants, and adjust the audience vs. control group sizes.
  3. For each variant, choose which message type to send and enter its content.
  4. Direct the users’ next step after they tap or swipe the message variants.
  5. Decide how and when you want the message variants delivered.
  6. Preview and send the message variants.

Features and options are explained along the way.

Steps

Choose your project from the Urban Airship dashboard, then click the Create button and select the A/B Test composer.


Audience

The audience you select in this step defines the entire set of users for the A/B test, which will include the control group (users who receive no message) and users who will receive different messages, according to the variants that you define.

  1. Select the radio button for your desired Audience.
    • All Devices sends the message to your entire audience.
    • Target Specific Users lets you create a recipient group based on segmentation data. See: Target Specific Users.
    • Test Devices is a predefined recipient group. See: Test Devices.
  2. Optionally modify which Platforms are included in your audience. All platforms configured for your project will be listed and are enabled by default, but you can toggle to limit your audience to specific platforms. You must select at least one.

  3. Click Variants in the header to move on.

Variants

  1. Enter a descriptive Test Name.
  2. Select the Number of Variants for your test, up to 26. The default is two. You will be able to add or delete variants in later steps.

  3. Use the slider to adjust the percentages of your target audience vs. control group. By default we send your test to 80% of your audience, keeping a control group of 20%

Content

Variant A Set Up

  1. Enter a name for the first Variant.

  2. Choose between sending a Push Notification or an In-App Message, or choose Silent Push Notification to send a push without notification text. After you make your selection you have the option to combine message types, then click the Continue button to confirm your choice(s).

  3. Enter the text that will display in your message. A preview will display as you type.
  4. If you chose to combine a Push Notification and In-App Message, an In-App Message Text section appears below the message truncation indicator. By default, the In-App message’s alert text is the same as the Push Notification text. If you’d prefer to use a different message, select the radio button for Write alternative, and enter your text in the box that appears below.
  5. Select a Message Action from the dropdown menu.

    The Message Actions listed in the dropdown menu differ depending on which message type you chose and if you chose to combine message types. See:
    Message Actions: Availability.


  6. Optionally Set a tag.
  7. Set Optional Features.
  8. Configure additional variants. Select the next variant by using the lettered tabs, and complete the name and message setup for each.

    For variants B-Z, you have the option to either use the content from a previously created variant, or start from a blank message.

    You may also add and delete variants using the tab X and + options.

  9. Click Delivery in the header to move on.

Delivery

  1. Select the radio button for either Send Now or Send at a Specific Date and Time. If sending at a Specific Date at Time, make your selections from the dropdown menus.

  2. Set Optional Delivery Features.

  3. Click Review & Send in the header to move on. If you chose to schedule the message delivery, this step will instead be labeled Review & Schedule.

Review & Send

  1. Select a variant from the dropdown menu in the Content section or above the previewer.

  2. Review the information on the left side of the screen, and click through the images in the previewer. The platform and display type are dynamically updated above.
    If you would like to make changes, click the associated step in the progress header, make your changes, then return to Review & Send.

  3. Send your message! Click the Send or Schedule button at the bottom of the window.