> Full SurrealDB documentation index: https://surrealdb.com/docs/llms.txt # DEFINE ANALYZER In the context of a database, an analyzer plays a crucial role in text processing and searching. It is defined by its name, a set of tokenizers, and a collection of filters. > [!NOTE] > Before SurrealDB version 3.0.0, the `FULLTEXT ANALYZER` clause used the syntax `SEARCH ANALYZER`. In the context of a database, an analyzer plays a crucial role in text processing and searching. It is defined by its name, a set of tokenizers, and a collection of filters. The output of an analyzer can be experimented with by using the [`search::analyze()`](/docs/reference/query-language/functions/database-functions/search.md#searchanalyze) function. ## Requirements - You must be authenticated as a root, namespace, or database user before you can use the `DEFINE ANALYZER` statement. - [You must select your namespace and database](/docs/reference/query-language/statements/use.md) before you can use the `DEFINE ANALYZER` statement. ## Statement syntax **SurrealQL Syntax** ```syntax title="SurrealQL Syntax" DEFINE ANALYZER [ OVERWRITE | IF NOT EXISTS ] @name [ FUNCTION @function ] [ TOKENIZERS @tokenizers ] [ FILTERS @filters ] [ COMMENT @string ] ``` ## The `FUNCTION` clause The `FUNCTION` clause runs a preprocessing step on the initial input before tokenizers and filters run. The reference must be a **function path** (not a call - omit parentheses), and the function must take and return a `string`. You can use either: * a [`fn::`](/docs/reference/query-language/statements/define/function.md) user-defined function defined with `DEFINE FUNCTION`, or * a [`mod::`](/docs/reference/query-language/statements/define/module.md) function from a Surrealism [extension module](/docs/learn/extensions/plugins/overview.md) defined with `DEFINE MODULE` (requires the [`surrealism` experimental capability](/docs/reference/cli/surrealdb-cli/commands/start.md#experimental-capabilities)). ```surql DEFINE FUNCTION fn::backwardsify($input: string) -> string { $input.split('').fold('', |$a, $b| $b + $a); }; DEFINE ANALYZER backwards FUNCTION fn::backwardsify TOKENIZERS blank; search::analyze("backwards", "I like SurrealDB"); ``` A Surrealism module function works the same way once the module is registered: ```surql DEFINE ANALYZER custom FUNCTION mod::demo::alter_string TOKENIZERS class; ``` ```surql title="Output" [ 'BDlaerruS', 'ekil', 'I' ] ``` ## Tokenizers Tokenizers are responsible for breaking down a given text into individual tokens based on a set of instructions. There are a couple of tokenizers that can be used while defining an analyzer as seen below: ### `blank` The blank tokenizer breaks down a text into tokens by creating a new token each time it encounters a space, tab, or newline character. It's a straightforward way to split text into words or chunks based on whitespace. ```surql DEFINE ANALYZER example_blank TOKENIZERS blank; search::analyze("example_blank", "hello world"); ``` ```surql title="Output" [ 'hello', 'world' ] ``` ### `camel` The camel tokenizer is used for identifying and creating tokens when the next character in the text is uppercase. This is particularly useful for processing camelCase or PascalCase text, common in programming, to split them into meaningful words. ```surql DEFINE ANALYZER example_camel TOKENIZERS camel; search::analyze("example_camel", "helloWorld"); ``` ```surql title="Output" [ 'hello', 'World' ] ``` ### `class` The class tokenizer segments text into tokens by detecting changes (digit, letter, punctuation, blank) in the Unicode class of characters. It creates a new token when the character class changes, distinguishing between digits, letters, punctuation, and blanks. This allows for flexible tokenization based on character types. ```surql DEFINE ANALYZER example_class TOKENIZERS class; search::analyze("example_class", "123abc!XYZ"); ``` ```surql title="Output" [ '123', 'abc', '!', 'XYZ' ] ``` ### `punct` The punct tokenizer generates tokens by breaking the text whenever a punctuation character is encountered. It's suitable for tokenizing sentences or breaking text into smaller units based on punctuation marks. ```surql DEFINE ANALYZER example_punct TOKENIZERS punct; search::analyze("example_punct", "Hello, World!"); ``` ```surql title="Output" [ 'Hello', ',', 'World', '!' ] ``` ### `segment(language)` {#segmentlanguage} _(since v3.3.0)_ The segment tokenizer splits text into words using a morphological dictionary, for languages the tokenizers above cannot split correctly. Those tokenizers decide every boundary from the characters around it, which works for languages that separate words with spaces or case changes. Chinese and Japanese write a whole clause with no separator at all, so `blank`, `class`, `camel` and `punct` all reduce one to a single token. Korean does use spaces, but attaches grammatical particles to the word they follow, so `한국어를` ("Korean" + object marker) stays whole and a search for `한국어` never matches it. `segment` takes one of `chinese`, `japanese` or `korean`: ```surql DEFINE ANALYZER example_korean TOKENIZERS blank,segment(korean); search::analyze("example_korean", "한국어를 배우고 있습니다"); ``` ```surql title="Output" [ '한국어', '를', '배우', '고', '있', '습니다' ] ``` The noun `한국어` ("Korean") is now its own token, so it matches independently of the particle attached to it. Korean written with Hanja is handled as well as Hangul, which matters for older newspapers, legal text and academic writing. The Hanja words are in the dictionary, and a Hangul particle attached to a Hanja stem separates from it: ```surql DEFINE ANALYZER example_plain TOKENIZERS blank,class; search::analyze("example_plain", "政府는 昨日 臨時國務會議를 召集하고"); search::analyze("example_korean", "政府는 昨日 臨時國務會議를 召集하고"); ``` ```surql title="Output" [ '政府는', '昨日', '臨時國務會議를', '召集하고' ] [ '政府', '는', '昨日', '臨時', '國務', '會議', '를', '召集', '하', '고' ] ``` `政府` and `會議` separate from the particles that follow them, and `臨時國務會議` separates into the three words it is built from, so each is searchable on its own. The dictionary covers modern standard Korean. Archaic Hangul written with the old jamo, including the arae-a (`ㆍ`) still used for Jeju, has no entries and is returned whole: ```surql search::analyze("example_korean", "ᄒᆞᆫ"); ``` ```surql title="Output" [ 'ᄒᆞᆫ' ] ``` Such text is not separated into words, though a boundary between an archaic sequence and a modern syllable is still found, because the two are written with different Unicode blocks: ```surql search::analyze("example_korean", "한ᄒᆞᆫ글"); ``` ```surql title="Output" [ '한', 'ᄒᆞᆫ글' ] ``` A longer sentence shows how much is at stake. Without `segment`, the space-separated tokenizers keep every word joined to the particle that follows it: ```surql DEFINE ANALYZER example_spaces TOKENIZERS blank,class; search::analyze("example_spaces", "서울에서 부산까지 기차를 타고 갑니다"); ``` ```surql title="Output" [ '서울에서', '부산까지', '기차를', '타고', '갑니다' ] ``` Neither `서울` nor `부산` is a token, so neither city can be searched for. `segment(korean)` separates each noun from its particle: ```surql search::analyze("example_korean", "서울에서 부산까지 기차를 타고 갑니다"); ``` ```surql title="Output" [ '서울', '에서', '부산', '까지', '기차', '를', '타', '고', '갑니다' ] ``` Japanese (with the small exception of spaced kana-only text for children or language learners) and Chinese need no other tokenizer, since there are no spaces to split on first: ```surql DEFINE ANALYZER example_japanese TOKENIZERS segment(japanese); search::analyze("example_japanese", "東京都に住んでいます"); ``` ```surql title="Output" [ '東京', '都', 'に', '住ん', 'で', 'い', 'ます' ] ``` Longer text separates in the same way, including place names written as compounds: ```surql search::analyze("example_japanese", "京都駅から大阪駅まで電車で行きます"); ``` ```surql title="Output" [ '京都', '駅', 'から', '大阪', '駅', 'まで', '電車', 'で', '行き', 'ます' ] ``` `京都駅` separates into the city and the station, so a search for `京都` reaches this text. One dictionary is used per language, but the search over it is the same for all three, so Japanese and Korean separate along the same joins. The sentence above and its Korean translation come apart morpheme for morpheme: ```surql DEFINE ANALYZER example_korean TOKENIZERS blank,segment(korean); search::analyze("example_korean", "도쿄도에 살고 있습니다"); ``` ```surql title="Output" [ '도쿄', '도', '에', '살', '고', '있', '습니다' ] ``` | Japanese | Korean | | | --- | --- | --- | | `東京` | `도쿄` | Tokyo | | `都` | `도` | metropolis | | `に` | `에` | location | | `住ん` | `살` | live | | `で` | `고` | joins the two verbs | | `い` | `있` | continuing state | | `ます` | `습니다` | polite ending | Both languages attach grammatical endings to a stem, and the segmenter separates each ending into a token of its own. That is why a search for the stem alone finds either sentence. The Japanese dictionary is modern, so older kana is not separated reliably. The kana iteration mark `ゝ`, which stands for a repeat of the kana before it, has no entry of its own and breaks the word it sits in: ```surql search::analyze("example_japanese", "こころ"); search::analyze("example_japanese", "こゝろ"); ``` ```surql title="Output" [ 'こころ' ] [ 'こ', 'ゝ', 'ろ' ] ``` Both spell the same word. Pre-1946 spellings, the obsolete kana `ゐ` and `ゑ`, and hentaigana vary in the same way: each is returned whole where nothing matches it, and broken into single kana where the pieces happen to match something else. The kanji iteration mark `々` is not affected, because words written with it, such as `人々` and `時々`, are entries in their own right. **Simplified** ```surql DEFINE ANALYZER example_chinese TOKENIZERS segment(chinese); search::analyze("example_chinese", "我喜欢数据库"); ``` ```surql title="Output" [ '我', '喜欢', '数据库' ] ``` **Traditional** ```surql DEFINE ANALYZER example_chinese TOKENIZERS segment(chinese); search::analyze("example_chinese", "我喜歡數據庫"); ``` ```surql title="Output" [ '我', '喜歡', '數據庫' ] ``` The dictionary holds both writings, so either segments correctly on its own. Japanese text read with `segment(chinese)` partly works, which can be misleading. Characters Japanese shares with Chinese are found, and Japanese-only forms fall out as single characters because no dictionary entry contains them. Chinese writes ice as `冰` in both Simplified and Traditional, where Japanese writes `氷`: ```surql search::analyze("example_chinese", "冰水"); search::analyze("example_chinese", "氷水"); ``` ```surql title="Output" [ '冰水' ] [ '氷', '水' ] ``` Both mean iced water, and `氷水` is a single word to `segment(japanese)`. Over a longer phrase the effect is uneven, which is what makes it hard to notice. Here the words Chinese shares are recovered and the ones holding a Japanese-only character are not: ```surql search::analyze("example_japanese", "北海道氷河時代地図"); search::analyze("example_chinese", "北海道氷河時代地図"); ``` ```surql title="Output" [ '北海道', '氷河', '時代', '地図' ] [ '北海道', '氷', '河', '時代', '地', '図' ] ``` `北海道` and `時代` are written the same way in both languages and survive. `氷河` and `地図` do not, because Chinese writes those characters as `冰` and `圖`, so neither compound is in the dictionary and each falls apart. Half the phrase is still searchable, which is why the wrong dictionary can pass a quick check. Use the dictionary for the language the text is written in. Chinese has no counterpart to that hiragana exception, and the reason is the reverse of what it suggests. Japanese marks its word boundaries by changing script: kanji write the content words and hiragana the grammar around them, so the switch from one to the other often shows where a word ends. Text written only in hiragana loses that and has to be spaced instead. Chinese never had it, because everything is written in one script, so there is nothing to take away. A Chinese reader separates the words by knowing them, which is the same thing the dictionary does here. Text for learners therefore annotates the characters rather than spacing them: zhuyin in Taiwan, pinyin in mainland China. Pinyin is written word by word, so `blank` already separates it and `segment` has nothing to add. Where both are available, the two agree on ordinary prose: `我在图书馆学习数据库设计` segments into the same six words that pinyin writes as `Wǒ zài túshūguǎn xuéxí shùjùkù shèjì`. They part company on the names of things. Pinyin writes the parts of a proper name separately, while the dictionary holds the whole name as one entry: | | `segment(chinese)` | pinyin | | --- | --- | --- | | `北京大学` | `北京大学` | `Běijīng Dàxué` | | `中华人民共和国` | `中华人民共和国` | `Zhōnghuá Rénmín Gònghéguó` | A search for `北京` therefore does not reach a document containing `北京大学`. Index the shorter name as well where both should match. Zhuyin is not in the dictionary, so a run of it is returned whole: ```surql search::analyze("example_chinese", "ㄋㄧˇㄏㄠˇ"); ``` ```surql title="Output" [ 'ㄋㄧˇㄏㄠˇ' ] ``` Text stored as zhuyin is therefore not searchable a word at a time. Spacing it does not recover the words either, because zhuyin is written one group per syllable while a word may be several syllables long. `数据库` is one word of three: ```surql search::analyze("example_chinese", "ㄕㄨˋ ㄐㄩˋ ㄎㄨˋ"); ``` ```surql title="Output" [ 'ㄕㄨˋ', 'ㄐㄩˋ', 'ㄎㄨˋ' ] ``` Word boundaries are decided by the dictionary rather than by character count, so words of one, two and three characters separate from each other in the same sentence: **Simplified** ```surql search::analyze("example_chinese", "我在图书馆学习数据库设计"); ``` ```surql title="Output" [ '我', '在', '图书馆', '学习', '数据库', '设计' ] ``` **Traditional** ```surql search::analyze("example_chinese", "我在圖書館學習數據庫設計"); ``` ```surql title="Output" [ '我', '在', '圖書館', '學習', '數據庫', '設計' ] ``` An analyzer may declare at most one `segment`. It runs after the other tokenizers, splitting the pieces they produced, so combining it with `blank` or `class` is useful for text that mixes scripts. #### Dictionaries Each language is segmented with its own dictionary. A dictionary holds the words of one language rather than the characters of one script, which matters because Han characters are used to write several languages. `segment(chinese)` knows Chinese words, not Han characters in general, so text in another language that borrows the script is read as though it were Chinese. Historical Vietnamese, written in Chữ Nôm, shows what that produces. Characters borrowed from Chinese are found, because they are Chinese words, while characters invented for Vietnamese have no entries and run together: ```surql search::analyze("example_chinese", "越南"); search::analyze("example_chinese", "𠬠𠊛"); ``` ```surql title="Output" [ '越南' ] [ '𠬠𠊛' ] ``` The first is found as the Chinese word for Vietnam. The second is two Vietnamese words, `một người` ("one person"), returned as a single token. Spacing would not recover them either: Vietnamese separates syllables rather than words, so `blank` would give syllables in the same way it does for pinyin. A dictionary covers the character forms it was built from. The Chinese dictionary holds both Simplified and Traditional writings, so each segments correctly, but it treats them as separate words rather than folding one into the other: ```surql search::analyze("example_chinese", "一样"); search::analyze("example_chinese", "一樣"); ``` ```surql title="Output" [ '一样' ] [ '一樣' ] ``` Both are the same word, and both produce a single token, but the tokens differ. Mixing the two writings in one piece of text is safe as long as each word is written consistently, because every word is looked up in the form it appears in. A word that mixes them internally matches nothing and falls back to single characters: ```surql search::analyze("example_chinese", "数據库"); ``` ```surql title="Output" [ '数', '據', '库' ] ``` `数据库` and `數據庫` are both in the dictionary; the half-converted `数據库` is not. This is worth knowing where text has been through an unreliable converter, because the result still reads correctly and only the tokens show the problem. A document indexed in Traditional characters is therefore not matched by a Simplified query, or the reverse. Where a collection mixes the two, convert to one form before indexing and convert queries the same way. The same applies to Japanese text written with older character forms. **The released binaries carry all three**, so `segment` works with nothing on disk and no configuration. That covers the downloads from [install.surrealdb.com](https://install.surrealdb.com), the Homebrew formula and the official Docker images, which package those same binaries. The dictionaries are the reason those downloads are substantially larger than they would otherwise be. **Replacing them from disk** is for the cases the shipped dictionaries do not cover: a custom or updated dictionary, or a build made without them. Set `SURREAL_SEGMENT_DICTIONARY_PATH` to a directory holding one subdirectory per language. The names are fixed, and each is the language it serves rather than the dictionary behind it: `korean`, `japanese`, `chinese`. Only the languages you use need to be present: **Bash** ```bash # /opt/surreal/dicts needs a subdirectory only for the languages it replaces; # the rest keep using the dictionaries built into the binary SURREAL_SEGMENT_DICTIONARY_PATH="/opt/surreal/dicts" \ surreal start --user root --pass secret ``` **PowerShell** ```powershell $env:SURREAL_SEGMENT_DICTIONARY_PATH = "C:\surreal\dicts" surreal start --user root --pass secret ``` The directory overrides the built-in dictionary for each language it carries, and leaves the rest alone. A directory holding only `korean` therefore replaces Korean while Japanese and Chinese keep using the dictionaries in the binary. Only an absent dictionary falls back that way. A dictionary that is present but cannot be read, and a directory named by `SURREAL_SEGMENT_DICTIONARY_PATH` that is not there at all, are errors rather than a silent fall back to the built-in one, so a misconfigured path is reported instead of quietly segmenting with something other than what you named. Not every build embeds the dictionaries. Builds from source do not unless the `cjk` feature is enabled, and neither does the WebAssembly package, which could not carry them. If the dictionary for a language can be found neither way, the `DEFINE ANALYZER` statement naming it fails, rather than being accepted and silently indexing unsegmented text. ## Filters Filters take on the task of transforming these tokens for further processing and analysis. ### `ascii` The ascii filter is responsible for processing tokens by replacing or removing diacritical marks (accents and special characters) from the text. It helps standardize text by converting accented characters to their basic ASCII equivalents, making it more suitable for various text analysis tasks. ```surql DEFINE ANALYZER example_ascii TOKENIZERS class FILTERS ascii; search::analyze("example_ascii", "résumé café"); ``` ```surql title="Output" [ 'resume', 'cafe' ] ``` ### `lowercase` The lowercase filter converts tokens to lowercase, ensuring that text is consistently in lowercase format. This is often used to make text case-insensitive for search and analysis purposes. ```surql DEFINE ANALYZER example_lowercase TOKENIZERS class FILTERS lowercase; search::analyze("example_lowercase", "Hello World"); ``` ```surql title="Output" [ 'hello', 'world' ] ``` ### `uppercase` The uppercase filter converts tokens to uppercase, ensuring text consistency in uppercase format. It can be useful when case-insensitivity is required for specific analysis or search operations. For example, if you had the text **"Hello World"**, the uppercase filter would create two tokens, **["HELLO", "WORLD"]**. Below is an example of how to use the uppercase filter: ```surql DEFINE ANALYZER example_uppercase TOKENIZERS class FILTERS uppercase; search::analyze("example_uppercase", "Hello World"); ``` ```surql title="Output" [ 'HELLO', 'WORLD' ] ``` ### `edgengram(min,max)` The edgengram filter is used to create tokens that represent prefixes of terms. It generates a sequence of tokens that gradually build up a term, which can be useful for autocomplete or searching based on partial words. It accepts two parameters `min` and `max` which define the minimum and maximum amount of characters in the prefix. For example, if you had the text **"apple banana"**, the edgengram filter would create six tokens, **["a", "ap", "app", "b", "ba", "ban"]**. Below is an example of how to use the edgengram filter: ```surql DEFINE ANALYZER example_edgengram TOKENIZERS class FILTERS edgengram(1,3); search::analyze("example_edgengram", "apple banana"); ``` ```text [ 'a', 'ap', 'app', 'b', 'ba', 'ban' ] ``` ### `mapper(path)` {#mapperpath} The mapping filter is designed to enable lemmatization within SurrealDB. Lemmatization is the process of reducing words to their base or dictionary form. The mapper mechanism allows users to specify a custom dictionary file that maps terms to their base forms. This dictionary file is then used by SurrealDB’s analyzer to standardize terms as they are indexed, improving search consistency. This is particularly useful for handling irregular verbs and other terms that the default "snowball" filter cannot handle. Lemmatization files are easy to put together and to find online, making it possible to customise full-text search for smaller languages. #### Filesystem allowlist A `DEFINE ANALYZER` statement with `mapper('')` opens the dictionary file on the **host filesystem** when the analyzer is defined. Access is gated by [`SURREAL_FILE_ALLOWLIST`](/docs/reference/cli/surrealdb-cli/environment-variables.md#file-config), without which no paths are permitted. Set one or more directories before using `mapper()`: **Bash** ```bash # Colon-separated directories SURREAL_FILE_ALLOWLIST="/var/surreal/dicts:/opt/wordlists" surreal start --user root --pass secret ``` **PowerShell** ```powershell # Semicolon-separated directories $env:SURREAL_FILE_ALLOWLIST = "C:\dicts;D:\wordlists" surreal start --user root --pass secret ``` The path in `mapper()` must resolve to a file **under** an allowed directory. Paths outside the allowlist are rejected at `DEFINE ANALYZER` time. > [!NOTE] > This allowlist is for analyzer dictionary files only. The experimental [files](/docs/learn/schema-management/files/buckets.md) feature uses [`SURREAL_BUCKET_FOLDER_ALLOWLIST`](/docs/reference/cli/surrealdb-cli/environment-variables.md#file-config) instead. How does the mapper work? Configuration: In the SQL statement below, the mapper parameter is specified within the analyzer definition. This parameter points to the file that contains the term mappings for lemmatization. ```surql DEFINE ANALYZER lemme_english TOKENIZERS blank,class FILTERS lowercase,mapper( '../tests/data/lemmatization-en.txt' ); RETURN [ search::analyze("lemme_english", "He drove and swam"), ]; ``` ```surql title="Output" [ [ 'he', 'drive', 'and', 'swim' ] ] ``` Dictionary File Structure: The file specified in the mapper parameter must follow this format: - Each line contains a pair of terms separated by a tab. - The first term represents the canonical (base form) of the word. - The second term is the form to be mapped to this base form. Example file format: ```text drive driven drive drives drive driving drive drove swim swam swim swimming swim swims swim swum ``` Usage: When this analyzer is applied to a text, any word that matches the mapped term in the dictionary file will be replaced by its base form before indexing. This helps ensure consistency in search results by consolidating different forms of a word to a single, standardized entry. By using this custom dictionary-based mapper, you can control how irregular forms and other variations of terms are indexed, making search behaviour more predictable and comprehensive. The following example shows how lemmatization can be used to generate a list of words and their respective frequencies. Other notable functionalities in the example are the [`string::is_alpha()`](/docs/reference/query-language/functions/database-functions/string.md#stringis_alpha) function inside [`array::filter()`](/docs/reference/query-language/functions/database-functions/array.md#arrayfilter) to remove all non-alphabetic strings, the [`type::record()`](/docs/reference/query-language/functions/database-functions/type.md#typerecord) function to construct a record ID from two strings, and an [`UPSERT`](/docs/reference/query-language/statements/upsert.md) statement to create a record if one does not exist, or update it otherwise. ```surql DEFINE ANALYZER lemme_english TOKENIZERS blank,class FILTERS lowercase,mapper( '../tests/data/lemmatization-en.txt' ); LET $text = "The Wheel of Time turns, and Ages come and pass, leaving memories that become legend. Legend fades to myth, and even myth is long forgotten when the Age that gave it birth comes again. In one Age, called the Third Age by some, an Age yet to come, an Age long past, a wind rose in the Mountains of Mist. The wind was not the beginning. There are neither beginnings nor endings to the turning of the Wheel of Time. But it was a beginning."; LET $words = search::analyze("lemme_english", $text) .filter(|$c| $c.is_alpha()); FOR $word IN $words { UPSERT type::record("word", $word) SET frequency += 1; }; SELECT * FROM word WHERE frequency >=3 ORDER BY frequency DESC; ``` ```surql title="Output" [ { frequency: 8, id: word:the }, { frequency: 6, id: word:age }, { frequency: 4, id: word:a }, { frequency: 4, id: word:be }, { frequency: 4, id: word:of }, { frequency: 3, id: word:and }, { frequency: 3, id: word:come }, { frequency: 3, id: word:to } ] ``` A mapper can also be used for ad-hoc filtering, as long as the file referenced contains two single words separated by a tab. Take the following file for example: ```title="error_filter.txt" NOT_FOUND File_not_found NOT_FOUND Datei_nicht_gefunden NOT_FOUND Fichier_non_trouvé TIMEOUT Timed_out TIMEOUT Délai_expiré TIMEOUT Zeitüberschreitung ``` An analyzer that uses a single mapper filter can then use this lemmatizer to unify multilingual error messages into a single output. ```surql DEFINE ANALYZER error_filter FILTERS mapper('error_filter.txt'); LET $messages = ["File not found", "Datei nicht gefunden", "Zeitüberschreitung"] .map(|$word| $word.replace(' ', '_')) .join(' '); search::analyze("error_filter", $messages); ``` ```surql title="Output" [ 'NOT_FOUND', 'NOT_FOUND', 'TIMEOUT' ] ``` Example using the same mapper to search for errors in multiple languages: ```surql DEFINE ANALYZER error_filter FILTERS mapper('error_filter.txt'); DEFINE INDEX OVERWRITE errors ON TABLE error FIELDS message FULLTEXT ANALYZER error_filter; FOR $message IN ["File not found", "Datei nicht gefunden", "Zeitüberschreitung"] { CREATE error SET message = $message.replace(' ', '_'), at = time::now(); }; SELECT * FROM error WHERE message @@ "NOT_FOUND"; ``` ```surql title="Output" [ { at: d'2024-11-13T03:56:12.039252Z', id: error:acbc044syhnx54wzs3n9, message: 'File_not_found' }, { at: d'2024-11-13T03:56:12.043643Z', id: error:5ifxic9s750x24ts4zof, message: 'Datei_nicht_gefunden' } ] ``` ### `ngram(min,max)` The ngram filter is used to create a sequence of 'n' tokens from a given sample of text or speech. These items can be syllables, letters, words or base pairs according to the application. It accepts two parameters `min` and `max` which indicates that you want to create n-grams starting from min to size of max. ```surql DEFINE ANALYZER example_ngram TOKENIZERS class FILTERS ngram(1,3); search::analyze("example_ngram", "apple banana"); ``` ```surql title="Output" [ 'a', 'ap', 'app', 'p', 'pp', 'ppl', 'p', 'pl', 'ple', 'l', 'le', 'e', 'b', 'ba', 'ban', 'a', 'an', 'ana', 'n', 'na', 'nan', 'a', 'an', 'ana', 'n', 'na', 'a' ] ``` ### `snowball(language)` The snowball filter applies Snowball stemming to tokens, reducing them to their root form and converts the case to lowercase. The following supported languages can be passed as a parameter in snowball: Arabic, Danish, Dutch, English, French, German, Greek, Hungarian, Italian, Norwegian, Portuguese, Romanian, Russian, Spanish, Swedish, Tamil, Turkish. ```surql DEFINE ANALYZER english_snowball TOKENIZERS class FILTERS snowball(english); DEFINE ANALYZER german_snowball TOKENIZERS class FILTERS snowball(german); RETURN [ search::analyze("english_snowball", "Looking at some running cats") search::analyze("german_snowball", "Sollen wir was trinken gehen?") ]; ``` ```surql title="Output" [ [ 'look', 'at', 'some', 'run', 'cat' ], [ 'soll', 'wir', 'was', 'trink', 'geh', '?' ] ] ``` ## Using `IF NOT EXISTS` clause The `IF NOT EXISTS` clause can be used to define an analyzer only if it does not already exist. You should use the `IF NOT EXISTS` clause when defining an analyzer in SurrealDB if you want to ensure that the analyzer is only created if it does not already exist. If the analyzer already exists, the `DEFINE ANALYZER` statement will return an error. It's particularly useful when you want to safely attempt to define a analyzer without manually checking its existence first. On the other hand, you should not use the `IF NOT EXISTS` clause when you want to ensure that the analyzer definition is updated regardless of whether it already exists. In such cases, you might prefer using the `OVERWRITE` clause, which allows you to define a analyzer and overwrite an existing one if it already exists, ensuring that the latest version of the analyzer definition is always in use. ```surql -- Create an ANALYZER if it does not already exist DEFINE ANALYZER IF NOT EXISTS example TOKENIZERS blank; ``` ## Using `OVERWRITE` clause The `OVERWRITE` clause can be used to create an analyzer and overwrite an existing one if it already exists. You should use the `OVERWRITE` clause when you want to modify an existing analyzer definition. If the analyzer already exists, the `DEFINE ANALYZER` statement will overwrite the existing analyzer definition with the new one. ```surql -- Create an ANALYZER and overwrite if it already exists DEFINE ANALYZER OVERWRITE example TOKENIZERS blank; ``` ## More examples Examples on application of analyzers to indexes can be found in the documenation on [`DEFINE INDEX`](/docs/reference/query-language/statements/define/indexes.md) statement This example creates an analyzer that tokenizes text based on the class of characters and then applies the lowercase filter to the tokens. ```surql -- Creates a simple analyzer removing diacritics marks DEFINE ANALYZER ascii TOKENIZERS class FILTERS lowercase,ascii; ``` This example creates an analyzer specifically designed for processing English texts. ```surql -- Creates an analyzer suitable for English text DEFINE ANALYZER english TOKENIZERS class FILTERS snowball(english); ``` This example creates an analyzer specifically designed for auto-completion tasks. ```surql -- Creates an analyzer suitable for auto-completion. DEFINE ANALYZER autocomplete FILTERS lowercase,edgengram(2,10); ``` This example creates an analyzer specifically designed for source code analysis. ```surql -- Creates an analyzer suitable for source code analysis. DEFINE ANALYZER code TOKENIZERS class,camel FILTERS lowercase,ascii; ``` ## Removing analyzers [`REMOVE ANALYZER`](/docs/reference/query-language/statements/remove.md) fails while any full-text index still references the analyzer. Remove or redefine those indexes first, then remove the analyzer. `REMOVE ANALYZER IF EXISTS` does not bypass this check when the analyzer is still in use.