A/B test REST API

Overview

A/B testing enables seamless relevance tuning experimentation by measuring the impact they have on the end users.

Note: each endpoint targets a single index.

Domain

The Analytics API and A/B test API can be reached from multiple domains, each specific to a region. You should use the domain that matches the region where your analytics data is stored and processed. View your analytics region.

The following domains are available:

• United States: https://analytics.us.algolia.com
• Europe (Germany): https://analytics.de.algolia.com

Note: https://analytics.algolia.com is an alias of https://analytics.us.algolia.com.

Common Parameters

Every endpoint accepts the following set of parameters:

• X-Algolia-Application-Id: The appID. Can also be passed through HTTP headers.
• X-Algolia-API-Key: Auth token. Can also be passed through HTTP headers.

ACL

For adding, stopping, and deleting A/B tests, the X-Algolia-API-Key needs to have the editSettings ACL.

For getting and listing A/B tests, X-Algolia-API-Key needs to have the analytics ACL.

Notes on usage

Some important considerations to keep in mind.

• You will need to have both analytics and clickAnalytics enabled on your account.
• Currently the feature is limited to 2 concurrent variants (A and B). We cannot target the same index in the 2 variants.
• The engine will spread traffic for both variants depending on the A/B test configuration, so there is no modification to perform on the frontend.
• The ordering in which variants are defined is important; the first variant index will be the “control index” meaning that when targeting this index, some traffic will be automatically redirected to the other variant index (how often is determined by the allotment percentage). However, when targeting the second variant index, no redirect will happen.
• While running a test, the Analytics Dashboard will show the combined metrics for both indices (cf previous point). However, this doesn’t fully make sense for some metrics (like no results). We might iterate on this in the future.
• We do not provide a way to validate that the UX will work with both variants for now so users should handle this on their side. We therefore recommend running your B index in your current UI before starting the A/B test
• A/B test analytics will not provide a daily breakdown but just a global view. A daily breakdown is actually not very useful when comparing the variants.
• The full A/B test history will be kept until the app is deleted or a test is explicitly deleted.
• The ABTest status might be:
  • active: the Analytics created the A/B test and performed a successful request to the engine
  • stopped: the A/B test was stopped by a user: it was deleted from the engine but we have to keep the data for historical purposes
  • expired: the A/B test reached its end date and was automatically stopped. It is removed from the engine but the metadata/metrics are kept
  • failed: the A/B test creation failed

Validation Error

In case of error, the API will return a payload in the following format:

1
2
3
4
{
  “status”: 123,
  “message”:”The error message”
}

A/B test Endpoints

Quick Reference

Path: /2/abtests
HTTP Verb: POST
Required API Key: any key with the setSettings ACL

Description:

Creates a new A/B test with provided configuration.

You can set an A/B test on two different indices with different settings, or on the same index with different search parameters by providing a customSearchParameters setting on one of the variants.

Parameters:

1
2
3
4
5
6
7
8
9
10
11
12
[
  {
    "index": string // mandatory
    "trafficPercentage": int // mandatory
    "description": string // optional
  },
  {
    "index": string // mandatory
    "trafficPercentage": int // mandatory
    "description": string // optional
  }
]

1
2
3
4
5
6
7
8
9
10
11
12
13
[
  {
    "index": string // mandatory
    "trafficPercentage": int // mandatory
    "description": string // optional
  },
  {
    "index": string // mandatory
    "trafficPercentage": int // mandatory
    "customSearchParameters": object // mandatory for one of the variants
    "description": string // optional
  }
]

Errors:

• 400: Invalid signature or invalid request
• 401: AppID or API Key missing
• 402: Feature not enabled
• 403: Invalid credentials
• 404: index not found
• 422: Invalid payload, A/B test already running on one of the indices
• 500: Internal error (possible e.g. if insertion query fails or engine query fails)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
curl -X POST \
     -H "X-Algolia-API-Key: ${API_KEY}" \
     -H "X-Algolia-Application-Id: ${APPLICATION_ID}" \
     --data-binary '{
      "name": "Custom Ranking salesRank Test",
      "variants": [
        {
          "index": "atis-abtest-default",
          "description": "current production index",
          "trafficPercentage": 70
        },
        {
          "index": "atis-abtest-salesRank",
          "trafficPercentage": 30
        }
      ],
      "endAt": "2018-05-17T23:59:59Z"
    }' \
    "https://analytics.algolia.com/2/abtests"

Upon success, the response is 200 OK.

1
2
3
4
5
{
  "abTestID":78,
  "taskID":111887230,
  "index":"atis-abtest-default"
}

Create an A/B test with the second variant enabling personalization:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
curl -X POST \
     -H "X-Algolia-API-Key: ${API_KEY}" \
     -H "X-Algolia-Application-Id: ${APPLICATION_ID}" \
     --data-binary '{
      "name": "Custom Ranking salesRank Test",
      "variants": [
        {
          "index": "atis",
          "trafficPercentage": 70
        },
        {
          "index": "atis",
          "trafficPercentage": 30,
          "customSearchParameters": {
              "enablePersonalization": true
          }
        }
      ],
      "endAt": "2018-05-17T23:59:59Z"
    }' \
    "https://analytics.algolia.com/2/abtests"

Path: /2/abtests/{id}
HTTP Verb: GET
Required API Key: any key with the analytics ACL

Description:

Returns metadata and metrics for A/B test id.

Behaves in the same way as GET /2/abtests however the endpoint will return 403.

Parameters:

Errors:

• 401: AppID or API Key missing
• 403: Forbidden in case the API Key has no access to the main index for this A/B test.

1
2
3
4
curl -X GET \
     -H "X-Algolia-API-Key: ${API_KEY}" \
     -H "X-Algolia-Application-Id: ${APPLICATION_ID}" \
    "https://analytics.algolia.com/2/abtests/${AB_TEST_ID}"

Upon success, the response is 200 OK.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
{
  "abTestID": 78,
  "clickSignificance": 1,
  "conversionSignificance": 0.9937,
  "createdAt": "2018-05-15T17:52:15.644906Z",
  "endAt": "2018-05-17T23:59:59Z",
  "name": "Custom Ranking salesRank Test",
  "status": "stopped",
  "variants": [
    {
      "averageClickPosition": 0,
      "clickCount": 17115,
      "clickThroughRate": 0.19839107906664039,
      "conversionCount": 11508,
      "conversionRate": 0.13339670101658765,
      "description": "",
      "index": "atis-abtest-default",
      "noResultCount": 0,
      "searchCount": 86269,
      "trackedSearchCount": 86269,
      "trafficPercentage": 70,
      "userCount": 55501
    },
    {
      "averageClickPosition": 0,
      "clickCount": 7716,
      "clickThroughRate": 0.20869847452125934,
      "conversionCount": 5129,
      "conversionRate": 0.13872660391647734,
      "description": "",
      "index": "atis-abtest-salesRank",
      "noResultCount": 0,
      "searchCount": 36972,
      "trackedSearchCount": 36972,
      "trafficPercentage": 30,
      "userCount": 22694
    }
  ]
}

Path: /2/abtests
HTTP Verb: GET
Required API Key: any key with the analytics ACL

Description:

Fetch all existing A/B tests for App that are available for the current API Key. Returns an array of metadata and metrics.

Notes:

• This endpoint is paginated in the same way as other Analytics endpoints (i.e. using ?limit=10&offset=0).
• As this endpoint fetches the list of all ABTests and then filters out the one that are not authorized for the given key, we do not return an error in case no ABTest is authorized for the key. We would just return an empty list of ABTests.
• When no data has been processed, the metrics will be returned as null

Parameters:

Errors:

• 401: AppID or API Key missing

1
2
3
4
curl -X GET \
     -H "X-Algolia-API-Key: ${API_KEY}" \
     -H "X-Algolia-Application-Id: ${APPLICATION_ID}" \
    "https://analytics.algolia.com/2/abtests?offset=${OFFSET}&limit=${LIMIT}"

Upon success, the response is 200 OK.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
{
  "abtests": [
    {
      "abTestID": 78,
      "clickSignificance": 1,
      "conversionSignificance": 0.9937,
      "createdAt": "2018-05-15T17:52:15.644906Z",
      "endAt": "2018-05-17T23:59:59Z",
      "name": "Custom Ranking salesRank Test",
      "status": "stopped",
      "variants": [
        {
          "averageClickPosition": 0,
          "clickCount": 17115,
          "clickThroughRate": 0.19839107906664039,
          "conversionCount": 11508,
          "conversionRate": 0.13339670101658765,
          "description": "",
          "index": "atis-abtest-default",
          "noResultCount": 0,
          "searchCount": 86269,
          "trackedSearchCount": 86269,
          "trafficPercentage": 70,
          "userCount": 55501
        },
        {
          "averageClickPosition": 0,
          "clickCount": 7716,
          "clickThroughRate": 0.20869847452125934,
          "conversionCount": 5129,
          "conversionRate": 0.13872660391647734,
          "description": "",
          "index": "atis-abtest-salesRank",
          "noResultCount": 0,
          "searchCount": 36972,
          "trackedSearchCount": 36972,
          "trafficPercentage": 30,
          "userCount": 22694
        }
      ]
    },
    {
      "abTestID": 80,
      "clickSignificance": 0.8187000000000001,
      "conversionSignificance": 0.6645,
      "createdAt": "2018-05-15T17:55:36.358352Z",
      "endAt": "2018-05-17T23:59:59Z",
      "name": "Searchable Attribute Manufacturer test",
      "status": "stopped",
      "variants": [
        {
          "averageClickPosition": 0,
          "clickCount": 65131,
          "clickThroughRate": 0.22,
          "conversionCount": 42808,
          "conversionRate": 0.14459719641952373,
          "description": "",
          "index": "atis-abtest-default",
          "noResultCount": 0,
          "searchCount": 296050,
          "trackedSearchCount": 296050,
          "trafficPercentage": 90,
          "userCount": 185719
        },
        {
          "averageClickPosition": 0,
          "clickCount": 7309,
          "clickThroughRate": 0.22219857724813036,
          "conversionCount": 4785,
          "conversionRate": 0.14546725846658964,
          "description": "",
          "index": "atis-abtest-manufacturer",
          "noResultCount": 0,
          "searchCount": 32894,
          "trackedSearchCount": 32894,
          "trafficPercentage": 10,
          "userCount": 20019
        }
      ]
    }
  ],
  "count": 2,
  "total": 2
}

Path: /2/abtests/{id}/stop
HTTP Verb: POST
Required API Key: any key with the setSettings ACL

Description:

Marks the A/B test as stopped. At this point, the test is over and cannot be restarted. As a result, your application is back to normal: index A will perform as usual, receiving 100% of all search requests. Associated metadata and metrics are still stored.

Parameters:

Errors:

• 400: Invalid signature or invalid request
• 401: AppID or API Key missing
• 402: Feature not enabled
• 403: Invalid credentials
• 404: A/B test ID not found
• 409: A/B test was correctly deleted from the engine but the Analytics request failed (inconsistent state)
• 422: Invalid ID format
• 500: Internal error (possible e.g. if insertion query fails or engine query fails)

1
2
3
4
curl -X POST \
     -H "X-Algolia-API-Key: ${API_KEY}" \
     -H "X-Algolia-Application-Id: ${APPLICATION_ID}" \
    "https://analytics.algolia.com/2/abtests/#{AB_TEST_ID}/stop"

Upon success, the response is 200 OK.

Path: /2/abtests/{id}
HTTP Verb: DELETE
Required API Key: any key with the setSettings ACL

Description:

Deletes the A/B Test and removes all associated metadata & metrics.

Parameters:

Errors:

• 400: Invalid signature or invalid request
• 401: AppID or API Key missing
• 402: Feature not enabled
• 403: Invalid credentials
• 404: A/B test ID not found
• 409: A/B test was correctly deleted from the engine but the Analytics request failed (inconsistent state)
• 422: Invalid payload, A/B test already running on one of the indices
• 500: Internal error (possible e.g. if insertion query fails or engine query fails)

1
2
3
4
curl -X DELETE \
     -H "X-Algolia-API-Key: ${API_KEY}" \
     -H "X-Algolia-Application-Id: ${APPLICATION_ID}" \
    "https://analytics.algolia.com/2/abtests/${AB_TEST_ID}"

Upon success, the response is 200 OK.