Engage Message Reports

Message Reports evaluate the performance of individual messages. The format of a message report varies depending on how the message was created. A/B Tests have a higher-level statistical report along with a message report for each variant.

Within this document, the Navigation section shows you how to locate your message reports, followed by the Report Format.

Note

See the Reports Menu Guide for in-depth analysis of your use of Engage.

Navigation

Access message reports via the Messages menu, in Messages Overview and A/B Tests.

Note

See the Messages Menu Guide for full documentation

Messages Overview has message reports for messages that were created via the Message, Automation, and In-App Automation workflows.

Message reports are accessible from three tabs: History, Automation, and In-App Automation.


The History tab lists messages:

  1. Created via the Message workflow that were either delivered immediately or were scheduled and have already been delivered.

  2. Created via the In-App Automation workflow that have been Expired or Stopped for more than 14 days.


The Automation tab lists all Active and Paused messages created via the Automation workflow.


The In-App Automation tab lists messages created via the In-App Automation workflow.

Note

After 14 days, messages with Expired or Stopped status move from In-App Automation to History.


A/B Tests is where you can view and manage messages created via the A/B Test workflow. Message reports for A/B Tests are accessible from the History tab.

The History tab lists tests containing messages that were either delivered immediately or were scheduled and have already been delivered.


Report Actions

Click a report action icon at the end of a message row. In-App Automation and A/B Tests have an icon for the Full Report only.

Eye = Preview Report
Expands the row, displaying metrics from the message report. Click the icon again to collapse the preview. Available metrics vary per message type. Click the question mark icon next to each for its definition.

Click Expand All and Collapse All to expose and hide Preview Reports for all listed messages.

Histogram = View Full Report
Opens the full message report or A/B Test Report.
Tip

Metrics can take 10 - 15 minutes to update after our servers have received the event.

Report Format

Reports for messages created via the Message, A/B Test, Automation, and In-App Automation workflows, and API counterparts contain the following components.

The report header contains a panel of options in the upper right corner and links to report sections at the bottom. Additional information displays slightly differently depending on which workflow was used to create the message. Those differences are described for each workflow type.

Message

The header displays the message's notification text and creation date, time, and time zone. The options panel is a single Duplicate button.

Options Panel

  • Duplicate: Creates a copy of the message and prompts you to choose a workflow for the new message. You must complete the workflow steps in order to send the new message.

Automation

The header displays the message's notification text and creation date, time, and time zone, as well as the message status and Automation Limits status. The options panel is in the blue banner and has a date range filter and controls to change the status of, edit, or duplicate the message.

Automation Limits

Edit your App-level automation rule limits, which control over-notification to users who repeatedly meet automation criteria, capping automated messages per device per given time interval, e.g., a maximum of 4 messages every 10 hours.

Click the pencil icon to open your project’s Configuration settings. Click the Automation Limits tab, and make and save your desired changes. See Settings: Configuration: Automation Limits for detail.

Options Panel

  • Pause/Resume: Active messages have the option to Pause. Paused messages have the option to Resume.

  • Cancel: Deletes the message.

  • Edit: Opens the message for editing.

  • Duplicate: Creates a copy of the message and opens it for editing. You must complete the workflow steps in order to send the new message.

Filter

Make your selection from the date range dropdown menu. The report updates immediately after selection.

In-App Automation

The header displays the message's name and status. The options panel has controls to change the status of or duplicate the message.

Options Panel

Active messages have the option to Stop. Stopped messages have the option to Restart.

  • Stop: Marks the message as canceled. This is effectively the same as specifying an end date, immediately applying an end date of "now." Active messages can be canceled at any time.

  • Restart: Changes the status of a Canceled or Expired message to Active. See: Restart Message. This is effectively a shortcut to editing the message and either eliminating or specifying an end date. Expired messages can be restarted within a grace period of 14 days. After 14 days they can only be duplicated.

  • Edit: Opens the message for editing. Availability depends on message status. Active messages can be edited at any time. An Expired or Canceled message can be edited and its end date extended, making it active again, within a grace period of 14 days. After 14 days they can only be duplicated.

  • Duplicate: Creates a copy of the message and opens it for editing. Yo must complete the workflow steps in order to save the new message.

API Payload

Click the arrow on the right side of the page to expose the message's API Payload.

The content included in the API Payload is what we call a push object. The push object describes everything about a push notification, including the audience and push payload. For more details on the push object, see: API: Push Object.

Note

In-App Automation does not have a public API, so the API Payload is not available for those messages.

Performance

Latest Message Activity

Latest Message Activity shows the username, date, time, and activity associated with the most recent change to the message. Last Message Activity data is available for 90 days after message creation. Click Show all to open a modal window that displays all message activity. This is only an option for messages with more than one activity.


Statistics

Directly below Latest Message Activity are statistics panes. Click the question marks to see term definitions appear as tooltips. Note that Direct Engagements will not appear if the message included a send to an Open Channel.


Engagement

Engagement data is available for all messages sent via push. Click the Download CSV button to export the data. Reports created via Automation have a time range dropdown menu instead of the Download CSV button. The menu options are dependent on the lifetime of the message, and the report updates immediately after making a selection.

Note

Engagement data does not apply to or appear for In-App Automation.

The bar chart includes engagement data for 12 hours following a message being sent. By default, the chart displays Direct Engagement for the message sent, e.g., a direct open on a mobile app or a click on a web notification.

Check the boxes to see additional information:

  • Platform Breakdown: Displays the chart data broken down by platform, e.g., iOS, Android, and Web.

  • Attributed Engagement: Displays the message's total engagement attributed to the notification.

    For messages sent to mobile app platforms, this uses our Influenced Opens metric. For messages sent to a web browser, this uses our Attributed Web Sessions metric. See Definitions below.

The chart remembers your selected preferences. If you visit a message report and enable Platform Breakdown, the next report you view will default to this presentation as well.

Below the chart are metrics tables for Platforms, as well as In-App Message and Message Center (Rich Page), if included in the message.

  • Platform: Notification sends and engagement metrics by platform.

  • In-App Messages: Total number of message views by opt-in and opt-out status.

  • Message Center (Rich Page): Total number of message sends, views, and deletions.

Definitions

As with other terms on the page, click the question marks to see definitions appear as tooltips. Additional notable definitions:

In-App Message Send
The number of in-app and silent sends associated with the message. Silent sends are sent via Background Processing.

An in-app message Send is logged for these scenarios:

  1. An in-app message, with or without an associated push, is sent to an opted-out device. Because opted-out devices will only receive the in-app message, this is counted as an in-app message Send.

  2. An in-app message only is sent to an opted-in device.
    Note

    An in-app message combined with a push notification that is sent to an opted-in device will always be counted as a notification Send rather than an in-app message Send.

In-App Message View
The number of times an in-app message was displayed.

An in-app message View is logged when a user encounters an in-app message, whether the in-app message was sent by itself or in combination with a push notification. Either of the two scenarios that result in an in-app message Send being logged will most likely also result in a View on open, assuming a user opens the app before the in-app message expires.

Note

An in-app message combined with a push notification that is sent to an opted-in user can also result in a View; if the user opens the app without tapping the push notification, the in-app message will still display, counting as a View.

Attributed Web Sessions
The total number of sessions attributed to the push notification.

An attributed session is a session that occurs within a 12-hour window of a web push notification. Sessions are generated when a user directly visits the website with the Urban Airship Web SDK present, or by clicking a web notification that leads the visitor to the site. The page the user visits must have the Web SDK present in order to track sessions.

Note

If a user visits the web page within 30 minutes prior to when the web push notification arrives, and they:

  • Click the web push notification, a new attributed session will be generated.

  • DO NOT click the web push notification, an attributed session will be generated only if there has been 30 minutes of inactivity on the website.


Message Detail

Message Detail is the same as shown in the Review step of a workflow, so the content and display will vary depending on the workflow used to create the message.

Click the arrows to page through the various previews. The platform and display type dynamically update in the dropdown menu above. You can also select a preview directly from the dropdown menu.

Delivery by Time Zone

The Delivery by Time Zone table is included for messages delivered according to a user's local time zone.

Metrics for Alerting Sends and Engagement are provided for each time zone. Below the table is the the total number of time zones with zero sends; this represents the number of time zones where none of your message recipients are located.

Note

Send a message to a user's local time zone:

For example, a push notification scheduled for 9 a.m. will arrive for people on the east coast at 9 a.m. Eastern Time, in the midwest an hour later at 9 a.m. Central Time, then on the west coast two hours after that, at 9 a.m. Pacific Time.

Events

If you have Custom Events enabled, the Events pane displays a list of events attributed, both directly and indirectly to the message. Click Download CSV to export the data.

  • Event Name: Human-readable name assigned to a particular Custom Event.
  • Notification Attribution: Displays whether your event was directly or indirectly attributed to the message.
  • Location: The source from which the event originates.
  • Count: Number of instances of this event. % of Total Count: Count divided by the Total Count (listed on the bottom row).
  • Value: The value (monetary or otherwise) generated by the event.
  • Avg. Value: Value divided by Count.
Note

Reported events are also accessible via Connect and Insight. These events include:

Message Impressions
Message Dismissed
Buttons Clicked (as custom events)

A/B Tests

A/B Tests have a higher-level statistical report along with message report for each variant.

An A/B Test Report displays the test name along with the Sent date, time, time zone, and Total Audience count. Variants are listed in the top table, with the Control Group below. Click a Variant name to view its individual message report.

Click the question marks to see term definitions.


Data Export

To export engagement data for the entire A/B test, click the Download CSV button in the upper right corner.

Note

Engagement data is sent to Urban Airship as soon as it becomes available. Often data is delayed because of connectivity issues with a user's carrier, wifi, power, etc. You should wait at least 12 to 24 hours before acting on this data, to allow for potential lags.

The following data are included in the A/B Test CSV export:

FieldDefinition
Project NameYour project name
App KeyUnique authentication key for your app
Push identifierUnique identifier that refers to the entire push send. The push identifier will be the same for each variant within an A/B Test, with N/A for users in the control group who do not receive a notification.
Sent TimeUTC timestamp
Test NameUser-friendly name for your A/B Test
VariantSee: variant
AudienceDescription of the selected audience for the A/B Test. One of All Users, Target Specific Users, or Test Users.
Notification AlertAlert text for the given variant
Notifications Sent/Control Sample SizeAudience send counts per variant/control group size
Indirect OpensOpens that occur within 12 hours of the notification, regardless of attribution
Direct OpensOpens that are attributed directly to interacting with the notification
Influenced OpensSee: Reports: Influenced Opens


Event Tracking by Variant

If you are using Custom Events, the Event Tracking by Variant report tracks event conversions and associated values, broken out by variant or control group.

Click Download CSV in the Event Tracking by Variant pane to download the report.

The following data is included in the Event Tracking by Variant CSV file:

App Name App Key
Push Identifier
Test Name
Variant
Event Name
Notification Attribution
Location Count Value