A/B Tests

Use A/B Tests to experiment with different variations, or variants , of a message for a given audience, and provide engagement reporting data for each variant.

Urban Airship’s A/B Test feature supports up to 26 variants and a control group per test, and is supported on iOS, Android, and Amazon devices.

Create a New A/B Test

Choose your app from the Urban Airship dashboard, then click the Create New button, and select the A/B Test composer.

Your steps are tracked in the progress header. Completed steps display a green checkmark icon, showing you are ready to proceed. A red (!) icon appears if you need to fix anything. Click a step name to move ahead or to go back and edit.

Your drafts are saved automatically. If you need to leave the composer, click Exit from the right side of the progress header. Use the dropdown arrow to Revert changes to a saved test, or Delete the test. Your saved tests are in Messages » A/B Tests » Drafts.

Your test’s selected conditions are displayed in a Summary on the right side of the window. The Help tab displays information relevant to the current step. Hover over composer sections to expose a (?) icon. Click the icon, and the Help tab will update with that topic.

Audience

Define who will receive your message.

The audience you select in this step defines the entire set of users for the A/B test, which will include the control group (users who receive no message) and users who will receive different messages, according to the variants that you define.

  1. Select the radio button for your desired Audience.

    • All Devices sends the message to your entire audience.
    • Target Specific Users lets you create a recipient group based on segmentation data. See Target Specific Users.
    • Test Devices is a predefined recipient group. See Test Devices.

  2. Optionally modify which Platforms are included in your audience. All platforms configured for your app will be listed and are enabled by default, but you can toggle to limit your audience to specific platforms. You must select at least one.

Variants

Name your test, enter the number of Variants, and adjust the audience vs. control group sizes.

  1. Enter a descriptive Test Name so you can easily find it in your full lists of tests and results.

  2. Select the Number of Variants for your test, up to 26. The default is two. You will be able to add or delete variants in later steps.

  3. Use the slider to adjust the percentages of your target audience vs. control group. By default we send your test to 80% of your audience, keeping a control group of 20%

Content

For each variant, choose which message type to send, enter its content, and configure options.

Variant A Set Up

  1. Enter a name for the first Variant.

  2. Choose between sending a Push Notification or an In-App Message. After you make your selection you have the option to combine both message types. Click the Continue button to confirm your choice(s).

    Or choose Silent Push Notification to send a push without notification text.

    • Push Notifications alert users from outside of your app to key information (news, offers, features, etc.), and drive them back to engage with your app.
    • In-App Messages are banner notifications that appear inside of your app. Use them to engage with users as they browse your app, and reach users who have not opted in to push.

  3. Enter the text that will display in your message. As you type, a preview will display.

    Below the text box is a color-coded indicator, warning when your message may be truncated on some devices.


    • Green: Full length of message will appear in most cases
    • Orange: Message may be shortened on some devices
    • Red: Message will be shortened on all devices

    Best practice is to make your message as brief as possible, though some customers find success when a message truncates, teasing their audience to learn more. Testing is the best way to optimize engagement rates for your audience.

    If you chose to combine a Push Notification and In-App Message, you have the option of using the Push Notification text as the In-App Message’s alert text. If you’d prefer to use a different message, select the radio button for Write alternative, and enter your text in the box that appears below.


  4. Direct the user’s next step after they tap or swipe your message. The Message Actions listed in the dropdown menu differ depending on which message type you chose, and if you chose to combine message types. See the Message Actions topic guide for more information.

    Action for Push Notification only:

    Action for In-App Message only:

    Action for combined Push Notification and In-App Message only:

    • Home Screen (combined) opens the app’s Home Screen for a Push Notification, and dismisses the In-App Message.

    Actions common to both Push Notification and In-App Message:

    • Landing Page opens a rich page that appears until it is closed, and it is not stored in the user’s Message Center inbox. See Rich Content Editor for procedure.
    • Deep Link opens to a configured screen within your app or on the web.
    • Web Page opens to a web page or any valid-device level URL such as App Store or app protocol links.
    • Share prompts the user to share the message on social networks.

  5. Click the Set a tag button.

    You may remove an existing tag, or set and remove tags.

    Type ahead to find tags that exist in the system, or create a new tag.

    Tags let you dynamically track user interactions for follow-on retargeting campaigns by setting one or more tags when the user interacts with this push. For example, if you set a tag “responded-campaign1” then you can target them with another message at a later date, knowing they are active users. Alternatively, you can re-engage them with an automated message if they are inactive for a period of time.

Optional Features

Available options are dependent on which message types and platforms are selected.

Buttons

Add one or two Buttons to bring even more interactivity to your messaging. Buttons prompt a user to take specific actions within or outside of the applications. There are up to 35 predefined Notification buttons to choose from, depending on your SDK version. Trigger any Urban Airship message action when a button is clicked. See Settings: Configuration: Notification Buttons.

Enable via toggle, click Select buttons, either page through or type to narrow your search, and click the button(s) of your choice.

Custom Keys

Set Custom Keys, also known as key/value pairs, which allow you to pass extra data through your Push Notification payload for use by custom code in your app. Common uses of custom keys include passing additional campaign identifiers for analytics, passing user profile information to the device, controlling the look and feel of the app, providing image links, etc.

Enable via toggle, make your platform selection from the dropdown menu, then enter the key and value. Click the Add Another button for additional keys.

By default, a custom key is sent to Push Notifications on all platforms, but you can choose platform-specific keys as well if you your message is going out to more than one platform, e.g., one imageURL for iOS and another imageURL for Android.

Title

Enter a Title to create a heading that appears above the notification text on:

  • iOS Notification Center
  • Apple Watch Looks
  • Android and Amazon Notification Area/Drawer

Enable via toggle, then type the title in the text field.

Summary

Add a Summary line, supplemental text displayed with the primary notification message. Enable via toggle, then type the line in the text field.

  • iOS: Summary appears below the push notification title. See iOS 10 documentation. iOS 10 only.
  • Android and Amazon: Summary appears below the main notification text in most cases. This is the only visible text other than the title when Android Picture is visible in expanded mode, as the main notification text is suppressed. See Android documentation.

Media

Add Media to your notification. Toggle to enable.

iOS
  1. Enter the URL for the media to be displayed with the push notification. Image, animated gif, audio, and video are supported. See API reference for allowed file types and requirements. Ensure the URL will be accessible by your mobile audience. HTTPS is required.

  2. Check the box to Hide iOS thumbnail. By default, a thumbnail of the media is displayed in the message alert.

  3. Add Media Specific Content on iOS 10 devices. By default, iOS 10 devices will receive the Same push notification as pre-iOS 10 devices. If you would like to send a different message to iOS 10 devices, select the Write Alternative radio button to display the iOS 10 Push Notification box, and enter the alternative text.

Alternate Title and Summary fields will appear here only if the initial Title and Summary fields preceding Media have been enabled and filled out.

Android and Amazon

Enter the URL for the media to be displayed with the push notification. Only static images are supported. See API reference for "style". Ensure the URL will be accessible by your mobile audience. HTTPS is required.

If the iOS platform is enabled along with Android and/or Amazon, the URL field is labeled Picture for Android. The Picture for Android field is removed if a URL for a static image is entered in the initial URL field.

Upload Media

If your account has CDN enabled, you have the option to Upload media rather than entering a URL. Select the radio button for Upload, click the Upload Media button, and select the media file. Maximum file size is 2 MB.

Supported Media

  • Images: JPEG, GIF, PNG
  • Audio: AIFF, WAV, MP3, M4A
  • Video: AVI, MPEG, MPEG2, MP4

Contact Support if you are interested in enabling CDN media hosting.

iOS Only

Badge

Update the recipient device’s app Badge, the red numeric display on the app icon that typically indicates its number of unread messages.

By default you automatically increment the existing badge number, but alternatively you can specify the exact number, e.g., +3, +12, -3.

Enable via toggle, then select either Increment by 1 or Specify. Specify requires a value in the text field.

Sound

Play a custom Sound when the message is received. The file must be bundled with your app by your app developer.

Enable via toggle, then enter the name of the sound file, e.g., “default” or “beep.caf”.

Additional Variants

  1. Select the next variant by using the lettered tabs, and complete the name and message setup for each.

    For variants B-Z, you have the option to either use the content from a previously created variant, or start from a blank message.

    You may also add and delete variants using the tab X and + options.


Delivery

Choose when the message will be sent.

Select the radio button for either Send Now or Send at a Specific Date and Time.

If sending at a Specific Date at Time, make your selections from the dropdown menus.

Optional Features

Expiration

Use Expiration to discard your message if it hasn’t been delivered or displayed by a specific time. Toggle to enable, then select the radio button for either Duration or Date & Time.

For Duration, enter the number of minutes, hours, or days past your defined send time to expire the delivery.

For Date & Time, enter the date and time when delivery attempts should cease.

Push Notifications

If a device is online, the notification will be delivered immediately. If a device is offline, the push services for each platform (e.g., APNs for iOS and GCM for Android) preserve that notification and attempt to deliver it as soon as it comes online, up until the expiration date. There is no way to remove a delivered Push Notification.

In-App Messages

Since In-App Messages are delivered via push notifications, they will not be delivered past expiration date if the user has been offline. In addition, Urban Airship will not display an In-App Message past the expiration date.

For more information about notification expiry, see Expiry in our API reference.

iOS Only


Background Processing

Turn on Background Processing to use the content-available flag. This wakes the app and gives it some running time to perform work, such as downloading content for future display.

Data included in the Push Notification is available for Background Processing, e.g., to send URLs and then download that content. Enable via toggle.

Mutable Content

Enabling Mutable Content allows a push notification’s content to be changed or downloaded before delivery. Automatically enabled if media is defined in Content » Optional Features » Media. iOS 10 only.

Android & Amazon Only

Delay

Delay delivery of your message until GCM detects that the device is active. This is a potential way to increase the likelihood of the user engaging with the notification. Enable via toggle.

Collapse Key

Use a Collapse Key to suppress previous messages sent with the same key. Only the most recent message (using the same key) will be viewable.

Enable via toggle, then type the key in the text field.

Review & Send

Confirm your choices, and preview the variants per platform. If you chose to schedule the message delivery, this step will instead be labeled Review & Schedule.

  1. Select a variant from the dropdown menu in the Content section or above the previewer.

  2. Review the information on the left side of the screen, and click through the images in the previewer. The platform and display type are dynamically updated above.


    If you would like to make changes, click the associated step in the progress header, make your changes, then return to Review & Send.

  3. Send your message! Click the Send or Schedule button at the bottom of your window.

A/B Test Reports

See Reports for in-depth analysis of your use of Engage. To evaluate the performance of A/B Tests, from the Urban Airship dashboard, click the Messages dropdown menu, and select A/B Tests.


The view defaults to the History tab, which lists tests that you have already sent.


Click the Reports action icon at the end of a test row.

  • Histogram = View Full Report Opens the top-level A/B Test Report. Click your browser’s Back button to return to the History list.

See Messages: A/B Tests for additional information about this menu section.

Top-Level Report

Below the test name is the test’s Sent date, time, and time zone, as well as its Total Audience count.

Variants are listed in the top table, with the Control Group below.


Click the question mark next to a column header, and its definition will display.


Click a Variant name to view its individual message report.

Data Export

To export engagement data for the entire A/B test, click the Download CSV button in the upper right corner of the Top-Level Report.

Engagement data is sent to Urban Airship as soon as it becomes available. Often data is delayed because of connectivity issues with a user’s carrier, wifi, power, etc. You should wait at least 12 to 24 hours before acting on this data, to allow for potential lags.

The following data are included in the A/B Test CSV export:

Field Definition
App Name Your app name
App Key Unique authentication key for your app
Push identifier Unique identifier that refers to the entire push send. The push identifier will be the same for each variant within an A/B Test, with N/A for users in the control group who do not receive a notification.
Sent Time UTC timestamp
Test Name User-friendly name for your A/B Test
Variant See: variant
Audience Description of the selected audience for the A/B Test. One of All Devices, Target Specific Users, or Test Devices.
Notification Alert Alert text for the given variant
Notifications Sent/Control Sample Size Audience send counts per variant/control group size
Indirect Opens Opens that occur within 12 hours of the notification, regardless of attribution
Direct Opens Opens that are attributed directly to interacting with the notification
Influenced Opens See: influenced open


Event Tracking by Variant

If you are using Custom Events, the Event Tracking by Variant report tracks event conversions and associated values, broken out by variant or control group.

Click the Download CSV button in the Event Tracking by Variant pane.


The following data are included in the Event Tracking by Variant CSV export:

  • App Name App Key
  • Push Identifier
  • Test Name
  • Variant
  • Event Name
  • Notification Attribution
  • Location Count Value