Guides / Managing results / Optimize search results / Typo tolerance

Oct. 25, 2020

Searching in Hyphenated Attributes

Formatting records

Formatting non-alphanumeric characters

Searching through hyphenated attributes is tricky, and it’s not just because users search with different formats. It’s also tricky because these attributes include non-alphanumeric characters, such as hyphens (-), pound signs (#), or plus signs (+).

By default, Algolia doesn’t index non-alphanumeric characters, or “separators”, meaning they are not searchable. They are, however, essential for tokenization. To learn more about how we handle tokenization, please refer to our in-depth guide on tokenization.

For example, we tokenize the string 1234-XYZ-B5 as 1234, -, XYZ, -, B5, because the hyphen (-) is a separator, and all the other characters are not. Then, by default, we only index the non-separator tokens 1234, XYZ, B5. The same is true for the string 1234 XYZ B5, since a space is also a separator. That’s why 1234-XYZ-B5 and 1234 XYZ B5 are functionally the same for the engine. Both would be a match, whether your user searches for 1234-XYZ-B5 or 1234 XYZ B5.

Copy
{
  "serial_number": "1234-XYZ-B5"
}

Copy
{
  "serial_number": "1234 XYZ B5"
}

There is no need to index special characters, unless they distinguish important differences between records. For example, you should only do this if the results for 1234-XYZ-B5 must be different than the results or 1234 XYZ B5. This is very rarely the case for ISBN, SKU, phone number, or serial number use cases, but could be true for other use cases, such as when searching for the programming languages C vs. C++.

Include all possible formats

Difficulties can arise when users search with a format that removes spaces or special characters, for example, 1234XYZB5. While Algolia handles some splitting and concatenation, there are special considerations when numbers are involved, and we may not handle several concatenations at once. For that reason, we recommend indexing all possible formats your users search with, not counting using different separators.

For example, this indexing format includes all different formats, but only uses spaces and does not include any versions with hyphens:

Copy
{
  [...],
  "serial_number": [
    "1234 XYZ B5",
    "1234XYZB5",
    "1234XYZ B5",
    "1234 XYZB5",
    "1234 XYZ",
    "XYZ B5"
  ]
}

This format will make the record a match when users search for any of these variants:

1234-XYZ-B5
1234 XYZ B5
1234XYZB5
1234XYZ-B5
1234XYZ B5
1234-XYZB5
1234 XYZB5
1234-XYZ
1234 XYZ
XYZ-B5
XYZ B5

There is some redundancy in this formatting. We index the following tokens for each element:

Copy
{
  [...],
  "serial_number": [
    "1234 XYZ B5", // tokens: ["1234", "XYZ", "B5"]
    "1234XYZB5", // tokens: ["1234XYZB5"]
    "1234XYZ B5", // tokens: ["1234XYZ", "B5"]
    "1234 XYZB5", // tokens: ["1234", "XYZB5"]
    "1234 XYZ", // tokens: ["1234", "XYZ"]
    "XYZ B5", // tokens: ["XYZ", "B5"]
  ]
}

Since the unique tokens are ["1234", "XYZ", "B5", "1234XYZB5", "1234XYZ", "XYZB5"], these are the only ones that need indexing.

That said, you may not want to undertake the work necessary to de-duplicate redundant tokens. Additionally, having all variants allows for a more accurate proximity score, if your users tend to search with tokens in the same order as the original unformatted version. For example, while a user may search for 1234 XYZB5, they probably won’t search for XYZB5 1234.

Suffix search

By design, Algolia doesn’t support infix/suffix matching; therefore, we won’t find substrings in the middle or the end of a string. If you want users to be able to search on suffixes, you need to index them.

Copy
{
  [...],
  "serial_number": "1234XYZB5",
  "serial_number_suffixes": [
    "234XYZB5",
    "34XYZB5",
    "4XYZB5",
    "XYZB5",
    "YZB5",
    "ZB5",
    "B5",
  ]
}

Algolia is optimized to search in textual attributes. Therefore, make sure to store numerical attributes used for search (as opposed to filtering, ranking, or sorting) as strings, not numbers.

One format for search and another for display

If you want to display one particular format, you need to include another attribute for the display value.

Copy
{
  [...],
  "serial_number": [
    "1234 XYZ B5",
    "1234XYZB5",
    "1234XYZ B5",
    "1234XYZ B5",
    "1234 XYZ",
    "XYZ B5"
  ]
  "serial_number_for_display": "1234-XYZ-B5"
}

To make the response payload more lightweight, you can exclude the non-display version from the response using attributesToRetrieve.

Configuring the relevance

Configuring `searchableAttributes`

To ensure that various formats are searchable, you first need to add the formatted attribute to the searchableAttributes. These attributes can be either strings or arrays.

Copy

1
2
3
4
5
6
7
8
$index->setSettings(
  [
    'searchableAttributes' => [
      ...,
      'serial_number',
      ... ]
  ]
);

1
2
3
4
5
6
7
8
9
10
11
12
13
IndexSettings settings = new IndexSettings
{
  SearchableAttributes = new List<string> { 
    ...,
    "serial_number",
    ...
    }
};

index.SetSettings(setting);

// Asynchronous
await index.SetSettingsAsync(settings);

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
// Sync version
index.setSettings(
  new IndexSettings()
    .setSearchableAttributes(Arrays.asList(
      ...,
      "serial_number",
      ...
      ))
);

// Async version
index.setSettingsAsync(
  new IndexSettings()
    .setSearchableAttributes(Arrays.asList(
      ...,
      "serial_number",
      ...
      ))
);

// To add a custom setting for an index,
// extend the class `IndexSettings` and add your getter/setter.
// The new settings will be automatically serialized by jackson.

1
2
3
4
5
6
7
8
settings := search.Settings{
  SearchableAttributes: opt.SearchableAttributes(
    ...,
    "serial_number",
    ...
    ),
}
res, err := index.SetSettings(settings)

1
2
3
4
5
6
7
8
9
val result: Future[Task] = client.execute {
  changeSettings of "myIndex" `with` IndexSettings(
    searchableAttributes = Some(Seq(
      ...,
      SearchableAttributes.attributes("serial_number"),
      ...
    ))
  )
}

If you added alternative forms in a separate attribute, you should also include it in searchableAttributes.

Copy

1
2
3
4
5
6
7
8
9
$index->setSettings(
  [
    'searchableAttributes' => [
      ...,
      'serial_number',
      'serial_number_suffixes',
      ... ]
  ]
);

1
2
3
4
5
6
7
8
9
index.setSettingsAsync(
  new JSONObject().put("searchableAttributes", 
  new JSONArray()
    ...,
    .put("serial_number")
    .put("serial_number_suffixes")
    ...
  )
);

1
2
3
4
5
6
7
8
9
10
11
12
13
14
IndexSettings settings = new IndexSettings
{
  SearchableAttributes = new List<string> { 
    ...,
    "serial_number",
    "serial_number_suffixes",
    ...
    }
};

index.SetSettings(setting);

// Asynchronous
await index.SetSettingsAsync(settings);

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
// Sync version
index.setSettings(
  new IndexSettings()
    .setSearchableAttributes(Arrays.asList(
      ...,
      "serial_number",
      "serial_number_suffixes",
      ...
      ))
);

// Async version
index.setSettingsAsync(
  new IndexSettings()
    .setSearchableAttributes(Arrays.asList(
      ...,
      "serial_number",
      "serial_number_suffixes",
      ...
      ))
);

// To add a custom setting for an index,
// extend the class `IndexSettings` and add your getter/setter.
// The new settings will be automatically serialized by jackson.

1
2
3
4
5
6
7
8
9
settings := search.Settings{
  SearchableAttributes: opt.SearchableAttributes(
    ...,
    "serial_number",
    "serial_number_suffixes",
    ...
    ),
}
res, err := index.SetSettings(settings)

1
2
3
4
5
6
7
8
9
10
val result: Future[Task] = client.execute {
  changeSettings of "myIndex" `with` IndexSettings(
    searchableAttributes = Some(Seq(
      ...,
      SearchableAttributes.attributes("serial_number"),
      SearchableAttributes.attributes("serial_number_suffixes"),
      ...
    ))
  )
}

1
2
3
4
5
6
7
8
9
10
val settings = settings {
    searchableAttributes {
        +...
        +"serial_number"
        +"serial_number_suffixes"
        ...
    }
}

index.setSettings(settings)

If you added an additional display attribute, you should not include it in searchableAttributes.

Hyphenated attributes need to match exactly (without typos)

Often, when a user searches for a particular number, they are looking for only that record, and are not interested in close matches. In that case, you can disable typo tolerance on the hyphenated number attribute using disableTypoToleranceOnAttributes.

Copy

1
2
3
4
5
6
7
$index->setSettings(
  [
    'disableTypoToleranceOnAttributes' => [
      'serial_number',
      ... ]
  ]
);

1
2
3
4
5
6
7
8
9
10
11
12
IndexSettings settings = new IndexSettings
{
  DisableTypoToleranceOnAttributes = new List<string> { 
    "serial_number",
    ...
    }
};

index.SetSettings(setting);

// Asynchronous
await index.SetSettingsAsync(settings);

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
// Sync version
index.setSettings(
  new IndexSettings()
    .setDisableTypoToleranceOnAttributes(Arrays.asList(
      "serial_number",
      ...
      ))
);

// Async version
index.setSettingsAsync(
  new IndexSettings()
    .setDisableTypoToleranceOnAttributes(Arrays.asList(
      "serial_number",
      ...
      ))
);

// To add a custom setting for an index,
// extend the class `IndexSettings` and add your getter/setter.
// The new settings will be automatically serialized by jackson.

1
2
3
4
5
6
7
settings := search.Settings{
  DisableTypoToleranceOnAttributes: opt.DisableTypoToleranceOnAttributes(
    "serial_number",
    ...
    ),
}
res, err := index.SetSettings(settings)

1
2
3
4
5
6
7
8
val result: Future[Task] = client.execute {
  changeSettings of "myIndex" `with` IndexSettings(
    disableTypoToleranceOnAttributes = Some(Seq(
      "serial_number",
      ...
    ))
  )
}

Hyphenated attributes need to match entirely (without prefix)

Similarly, if you would like to display results only on complete (and not prefixed) matches, you can disable prefix matching on the hyphenated attribute using disablePrefixOnAttributes.

Copy

1
2
3
4
5
6
7
$index->setSettings(
  [
    'disablePrefixOnAttributes' => [
      'serial_number',
      ... ]
  ]
);

1
2
3
4
5
6
7
8
9
10
11
12
IndexSettings settings = new IndexSettings
{
  DisablePrefixOnAttributes = new List<string> { 
    "serial_number",
    ...
    }
};

index.SetSettings(setting);

// Asynchronous
await index.SetSettingsAsync(settings);

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
// Sync version
index.setSettings(
  new IndexSettings()
    .setDisablePrefixOnAttributes(Arrays.asList(
      "serial_number",
      ...
      ))
);

// Async version
index.setSettingsAsync(
  new IndexSettings()
    .setDisablePrefixOnAttributes(Arrays.asList(
      "serial_number",
      ...
      ))
);

// To add a custom setting for an index,
// extend the class `IndexSettings` and add your getter/setter.
// The new settings will be automatically serialized by jackson.

1
2
3
4
5
6
7
settings := search.Settings{
  DisablePrefixOnAttributes: opt.DisablePrefixOnAttributes(
    "serial_number",
    ...
    ),
}
res, err := index.SetSettings(settings)

1
2
3
4
5
6
7
8
val result: Future[Task] = client.execute {
  changeSettings of "myIndex" `with` IndexSettings(
    disablePrefixOnAttributes = Some(Seq(
      "serial_number",
      ...
    ))
  )
}

Handling non-alphanumeric characters

By default, the engine ignores non-alphanumeric characters like hyphen (-), plus (+), and parentheses ((,)). Whether these characters are in the query or the index, we won’t search for them.

This is by design: searching for +33 returns all records with an attribute starting with +33 or 33, because we ignore the plus sign (+). If you would like users to be able to search with special characters, you must let the engine know to index these characters. You can do so with separatorsToIndex.

For example, if you include + in separatorsToIndex, searching for +33 will only return records containing both + and 33. Since adding separatorsToIndex can make a search more restrictive and complex, we find that is generally not desirable to do so for these use cases.

We do not recommend including a character as a separatorsToIndex unless its presence distinguishes between different records. For example, if searches for 1234-XYZ-B5 and 1234 XYZ B5 should return different results. This is rarely true for SKU, ISBN, phone numbers, and serial number use cases.

Searching with `removeWordsIfNoResults` enabled

The setting removeWordsIfNoResults allows you to loosen the query when the engine initially doesn’t find any results. Because of the way we handle concatenation with non-alphanumeric characters, this parameter may not work as you expect when searching for hyphenated attributes.

Please refer to our guide on using removeWordsIfNoResults with non-alphanumeric characters for more information.

Building Search UI

Building Search UI

Building Search UI

Building Search UI

Building Search UI

Building Search UI

PHP

Ruby

JavaScript

Python

Swift

Kotlin

Android

.NET

Java

Golang

Scala

InstantSearch.js

React InstantSearch

Vue InstantSearch

Angular InstantSearch

InstantSearch iOS

InstantSearch Android

Autocomplete

Crawler Configuration API

Index settings and search parameters

A full reference of API Endpoints

Rails

Symfony

Django

Laravel

Crawler

Magento 2

WordPress

Shopify

Salesforce Commerce Cloud B2C

Netlify

Searching in Hyphenated Attributes

On this page

Formatting records

Formatting non-alphanumeric characters

Include all possible formats

Suffix search

One format for search and another for display

Configuring the relevance

Configuring searchableAttributes

Hyphenated attributes need to match exactly (without typos)

Hyphenated attributes need to match entirely (without prefix)

Handling non-alphanumeric characters

Searching with removeWordsIfNoResults enabled

Did you find this page helpful?

On this page

Configuring `searchableAttributes`

Searching with `removeWordsIfNoResults` enabled