In this use case, we will set up an A/B test to investigate how the sequence of delivery options impacts cart conversion rates. We will compare two checkout configurations with the same delivery options, only the order will be different.

• Configuration A: DHL Service Point on top
• Configuration B: PostNord MyPack Home on top

Learn how to:

1. Set up the Checkout configuration
2. Configure the experiment
3. Update your Connection to use an Experiment
4. View the test results

Set up the Checkout configuration

First, you need to configure the Checkout configuration that you want to compare to your default Checkout. The easiest way to do this is by duplicating your default configuration and making changes to the part you want to test.

1. In nShift Portal go to Checkout > Configurations and locate your default checkout.
   
   
2. Click on the three dots menu and choose Duplicate.
   
   
   
   
3. Update the Name of the configuration to differentiate it from the original configuration. The name is only for internal use and will be displayed in the Checkout’s configuration interface.
   
   
4. Select the member account where the Checkout configuration should be saved. 
   
   
5. Select a template and a sender. All other settings will be prefilled with the data from the original configuration. 
   
   
6. Our default configuration has DHL Service Point on top. Go to Delivery options and change the order of the delivery options by dragging and dropping them so PostNord MyPack Home is on top.
   
   
   
   
7. Click Save. 

Note: If you are setting up a Checkout configuration from scratch, you can find instructions in this article: Create an nShift Checkout configuration

Configure the experiment

We are now ready to configure the experiment. This is a simple setup because we only want to split the traffic between our two configurations. 

1. Go to Experiments in the top menu and click Create new experiment.
   
   
2. Fill in Name and choose the Member account from the drop-down list (should be the same as used for your checkout configurations).
   
   
3. Add a description to remember the purpose of the experiment.
   
   
4. Enter the Start and Stop date of the experiment. The experiment will only run during this period. When the current date is before or after the experiment period, the configuration or group defined in the connection will be used.
   
   
5. Click Add new condition and use the toggle switch to Disable condition.
   
   
6. Add the two checkout configurations that should be compared and set the traffic share to 50% for each.
   
   
   
   
7. Click Continue and Save. The experiment will run for the planned time.
   
   
8. If you are using our API remember to update your connection to point to the experiment (read more here).

Image showing the entire experiment setup:

The Checkout widget examples

Configuration A: DHL Service Point on top:

Configuration B: PostNord MyPack Home on top:

Update your Connection to use an Experiment

If you are using the Checkout API, your requests contain a Connection ID to identify your Checkout configuration. To use an Experiment in your current integration, you simply point your connection to an Experiment. This is easily done in the user interface and requires no code changes. 

1. Go to Connections in the top menu.
   
   
2. Click the three-dots menu next to the Connection used by your integration and choose Edit.
   
   
3. Select the Experiment to be used from the Experiment drop-down list. 
   
   
   
   
4. Click Save. The Connection ID will now point to the Experiment instead.

View the test results

When the experiment is completed, you can download a metrics file to analyze the results. 

1. Go to Experiments in the top menu.
   
   
2. Click the three-dot menu next to the experiment for which you want the results and choose Download Results.
   
   
   
   
3. Click the Export button in the pop-up. The dates for the experiment are prefilled but you can adjust them if needed. 
   
   
   
   
4. A CSV file will be generated soon with the data collected during the time period.

You can also see key metrics under Segmentations > Results on the experiment configuration page.

Metrics data in the report

Below is an overview of the metrics included in the CSV file. Each line in the file represents a checkout session and its data.

• Member Account ID
• Request ID
• Connection ID
• Group ID
• Checkout ID
• Checkout Version
• Receiver Postal Code
• Receiver Country Code
• Language Code
• Currency Code
• Option ID
• Option Name
• Option Index
• Source System
• Carrier ID
• Carrier Product ID
• Created
• Selected
• Shipment Count
• Total Request Price
• Total Checkout Price
• Total Addons Price
• Total Full Price