{"componentChunkName":"component---src-templates-post-jsx","path":"/blog/using-the-intl-segmenter-api/","result":{"data":{"mdx":{"body":"function _extends() { _extends = Object.assign || function (target) { for (var i = 1; i < arguments.length; i++) { var source = arguments[i]; for (var key in source) { if (Object.prototype.hasOwnProperty.call(source, key)) { target[key] = source[key]; } } } return target; }; return _extends.apply(this, arguments); }\n\nfunction _objectWithoutProperties(source, excluded) { if (source == null) return {}; var target = _objectWithoutPropertiesLoose(source, excluded); var key, i; if (Object.getOwnPropertySymbols) { var sourceSymbolKeys = Object.getOwnPropertySymbols(source); for (i = 0; i < sourceSymbolKeys.length; i++) { key = sourceSymbolKeys[i]; if (excluded.indexOf(key) >= 0) continue; if (!Object.prototype.propertyIsEnumerable.call(source, key)) continue; target[key] = source[key]; } } return target; }\n\nfunction _objectWithoutPropertiesLoose(source, excluded) { if (source == null) return {}; var target = {}; var sourceKeys = Object.keys(source); var key, i; for (i = 0; i < sourceKeys.length; i++) { key = sourceKeys[i]; if (excluded.indexOf(key) >= 0) continue; target[key] = source[key]; } return target; }\n\n/* @jsxRuntime classic */\n\n/* @jsx mdx */\nconst _frontmatter = {\n  \"title\": \"Using the Intl segmenter API\",\n  \"cover\": \"/blogs/segmenter/cover.svg\",\n  \"date\": \"2023-10-03\",\n  \"type\": [\"tutorial\"]\n};\nconst layoutProps = {\n  _frontmatter\n};\nconst MDXLayout = \"wrapper\";\nreturn function MDXContent(_ref) {\n  let {\n    components\n  } = _ref,\n      props = _objectWithoutProperties(_ref, [\"components\"]);\n\n  return mdx(MDXLayout, _extends({}, layoutProps, props, {\n    components: components,\n    mdxType: \"MDXLayout\"\n  }), mdx(\"p\", null, `The `, mdx(\"code\", _extends({\n    parentName: \"p\"\n  }, {\n    \"className\": \"language-text\"\n  }), `Intl`), ` API in browsers has a ton of functionality around editing and formatting of text and numbers. While filling in the `, mdx(\"a\", _extends({\n    parentName: \"p\"\n  }, {\n    \"href\": \"https://stateofhtml.com/\"\n  }), `State of HTML`), ` survey I came across one feature I hadn't seen before: the `, mdx(\"a\", _extends({\n    parentName: \"p\"\n  }, {\n    \"href\": \"https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Intl/Segmenter\"\n  }), `segmenter API`), `. Upon looking into it I found out that it allows you to break up text into segments based on the language of the text.`), mdx(\"p\", null, `This caught my attention because I recently implemented a `, mdx(\"a\", _extends({\n    parentName: \"p\"\n  }, {\n    \"href\": \"https://polypane.app/docs/measure-text-length/\"\n  }), `text measurement feature`), ` and I was very annoyed that I managed to only support western languages using latin characters and spaces to separate words.`), mdx(\"p\", null, `I spent a lot of time with npm packages, regexes and a lot of trial and error to try and support other languages but didn't end up with a solution for them.`), mdx(\"p\", null, `Enter the segmenter API.`), mdx(\"h2\", {\n    \"id\": \"how-we-use-the-segmenter-api-in-polypane\"\n  }, `How we use the segmenter API in Polypane`), mdx(\"p\", null, `Whenever you select text in  Polypane and right-click to open the context menu, we'll show you the characters, words and emoji in the text selection.`), mdx(\"img\", {\n    src: measuretext,\n    className: \"imgshadow\",\n    alt: \"Measure text length context menu in Polypane.\"\n  }), mdx(\"p\", null, `Use this to quickly check if you've written the right number of characters for a tweet, or if you've written the right number of words for a blog post.`), mdx(\"h2\", {\n    \"id\": \"setting-up-the-segmenter-api\"\n  }, `Setting up the segmenter API`), mdx(\"p\", null, `To use it, first you create a new segmenter. That takes two arguments, a language and a set of options.`), mdx(\"div\", {\n    \"className\": \"gatsby-highlight\",\n    \"data-language\": \"js\"\n  }, mdx(\"pre\", _extends({\n    parentName: \"div\"\n  }, {\n    \"className\": \"language-js\"\n  }), mdx(\"code\", _extends({\n    parentName: \"pre\"\n  }, {\n    \"className\": \"language-js\"\n  }), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token keyword\"\n  }), `const`), ` segmenter `, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token operator\"\n  }), `=`), ` `, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token keyword\"\n  }), `new`), ` `, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token class-name\"\n  }), `Intl`, mdx(\"span\", _extends({\n    parentName: \"span\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `.`), `Segmenter`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `(`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token string\"\n  }), `'en'`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `,`), ` `, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `{`), ` `, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token literal-property property\"\n  }), `granularity`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token operator\"\n  }), `:`), ` `, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token string\"\n  }), `'word'`), ` `, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `}`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `)`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `;`)))), mdx(\"p\", null, `The options object has two properties, both of which can be omitted:`), mdx(\"ul\", null, mdx(\"li\", {\n    parentName: \"ul\"\n  }, mdx(\"code\", _extends({\n    parentName: \"li\"\n  }, {\n    \"className\": \"language-text\"\n  }), `localeMatcher`), ` has a value of 'lookup' or 'best fit' (default) with 'lookup' using a specific matching algorithm (`, mdx(\"a\", _extends({\n    parentName: \"li\"\n  }, {\n    \"href\": \"https://tools.ietf.org/html/bcp47\"\n  }), `BCP 47`), `) and 'best fit' using a looser algorithm that tries to match the language as closely as possible from what's available from the browser or OS.`), mdx(\"li\", {\n    parentName: \"ul\"\n  }, mdx(\"code\", _extends({\n    parentName: \"li\"\n  }, {\n    \"className\": \"language-text\"\n  }), `granularity`), ` has a value of 'grapheme' (default), 'word' or 'sentence'. This determines what types of segments are returned. Grapheme is the default and are the characters as you see them.`)), mdx(\"p\", null, `Then, you call `, mdx(\"code\", _extends({\n    parentName: \"p\"\n  }, {\n    \"className\": \"language-text\"\n  }), `segmenter.segment`), ` with the text you want to segment. This returns an iterator, which you can convert to an array using `, mdx(\"code\", _extends({\n    parentName: \"p\"\n  }, {\n    \"className\": \"language-text\"\n  }), `Array.from`), `:`), mdx(\"div\", {\n    \"className\": \"gatsby-highlight\",\n    \"data-language\": \"js\"\n  }, mdx(\"pre\", _extends({\n    parentName: \"div\"\n  }, {\n    \"className\": \"language-js\"\n  }), mdx(\"code\", _extends({\n    parentName: \"pre\"\n  }, {\n    \"className\": \"language-js\"\n  }), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token keyword\"\n  }), `const`), ` segments `, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token operator\"\n  }), `=`), ` segmenter`, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `.`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token function\"\n  }), `segment`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `(`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token string\"\n  }), `'This has four words!'`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `)`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `;`), `\nArray`, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `.`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token function\"\n  }), `from`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `(`), `segments`, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `)`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `.`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token function\"\n  }), `map`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `(`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `(`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token parameter\"\n  }), `segment`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `)`), ` `, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token operator\"\n  }), `=>`), ` segment`, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `.`), `segment`, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `)`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `;`), `\n`, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token comment\"\n  }), `// ['This', ' ', 'has', ' ', 'four', ' ', 'words', '!']`)))), mdx(\"p\", null, `Each item in the iterator is an object, and it has different values depending on the granularity but at least:`), mdx(\"ul\", null, mdx(\"li\", {\n    parentName: \"ul\"\n  }, mdx(\"code\", _extends({\n    parentName: \"li\"\n  }, {\n    \"className\": \"language-text\"\n  }), `segment`), ` is the actual text of the segment.`), mdx(\"li\", {\n    parentName: \"ul\"\n  }, mdx(\"code\", _extends({\n    parentName: \"li\"\n  }, {\n    \"className\": \"language-text\"\n  }), `index`), ` is the index of the segment in the original text, e.g. where it starts.`), mdx(\"li\", {\n    parentName: \"ul\"\n  }, mdx(\"code\", _extends({\n    parentName: \"li\"\n  }, {\n    \"className\": \"language-text\"\n  }), `input`), ` is the original text.`)), mdx(\"p\", null, `For the word granularity, it also has:`), mdx(\"ul\", null, mdx(\"li\", {\n    parentName: \"ul\"\n  }, mdx(\"code\", _extends({\n    parentName: \"li\"\n  }, {\n    \"className\": \"language-text\"\n  }), `isWordLike`), ` is a boolean that is true if the segment is a word or word-like.`)), mdx(\"p\", null, `Sp you might have looked at the array and said \"well those white spaces aren't words, are they?\". And you'd be right. To get the words, we need to filter the array:`), mdx(\"div\", {\n    \"className\": \"gatsby-highlight\",\n    \"data-language\": \"js\"\n  }, mdx(\"pre\", _extends({\n    parentName: \"div\"\n  }, {\n    \"className\": \"language-js\"\n  }), mdx(\"code\", _extends({\n    parentName: \"pre\"\n  }, {\n    \"className\": \"language-js\"\n  }), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token keyword\"\n  }), `const`), ` segments `, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token operator\"\n  }), `=`), ` segmenter`, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `.`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token function\"\n  }), `segment`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `(`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token string\"\n  }), `'This has four words!'`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `)`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `;`), `\nArray`, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `.`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token function\"\n  }), `from`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `(`), `segments`, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `)`), `\n  `, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `.`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token function\"\n  }), `filter`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `(`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `(`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token parameter\"\n  }), `segment`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `)`), ` `, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token operator\"\n  }), `=>`), ` segment`, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `.`), `isWordLike`, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `)`), `\n  `, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `.`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token function\"\n  }), `map`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `(`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `(`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token parameter\"\n  }), `segment`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `)`), ` `, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token operator\"\n  }), `=>`), ` segment`, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `.`), `segment`, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `)`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `;`), `\n`, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token comment\"\n  }), `// ['This', 'has', 'four', 'words']`)))), mdx(\"p\", null, `And now we know how many actual `, mdx(\"em\", {\n    parentName: \"p\"\n  }, `words`), ` that sentence has.`), mdx(\"p\", null, `Instead of words like the example above, you can also ask it to segment sentences, and it knows which periods end sentences and which are used for e.g. thousand separators in numbers:`), mdx(\"div\", {\n    \"className\": \"gatsby-highlight\",\n    \"data-language\": \"js\"\n  }, mdx(\"pre\", _extends({\n    parentName: \"div\"\n  }, {\n    \"className\": \"language-js\"\n  }), mdx(\"code\", _extends({\n    parentName: \"pre\"\n  }, {\n    \"className\": \"language-js\"\n  }), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token keyword\"\n  }), `const`), ` segmenter `, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token operator\"\n  }), `=`), ` `, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token keyword\"\n  }), `new`), ` `, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token class-name\"\n  }), `Intl`, mdx(\"span\", _extends({\n    parentName: \"span\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `.`), `Segmenter`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `(`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token string\"\n  }), `'en'`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `,`), ` `, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `{`), ` `, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token literal-property property\"\n  }), `granularity`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token operator\"\n  }), `:`), ` `, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token string\"\n  }), `'sentence'`), ` `, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `}`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `)`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `;`), `\n`, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token keyword\"\n  }), `const`), ` segments `, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token operator\"\n  }), `=`), ` segmenter`, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `.`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token function\"\n  }), `segment`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `(`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token string\"\n  }), `'This is a sentence with the value 100.000 in it. This is another sentence.'`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `)`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `;`), `\n\nArray`, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `.`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token function\"\n  }), `from`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `(`), `segments`, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `)`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `.`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token function\"\n  }), `map`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `(`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `(`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token parameter\"\n  }), `segment`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `)`), ` `, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token operator\"\n  }), `=>`), ` segment`, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `.`), `segment`, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `)`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `;`), `\n`, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token comment\"\n  }), `// [`), `\n`, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token comment\"\n  }), `//    'This is a sentence with the value 100.000 in it.',`), `\n`, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token comment\"\n  }), `//    'This is another sentence.'`), `\n`, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token comment\"\n  }), `// ]`)))), mdx(\"p\", null, `There are no `, mdx(\"code\", _extends({\n    parentName: \"p\"\n  }, {\n    \"className\": \"language-text\"\n  }), `isSentenceLike`), ` and `, mdx(\"code\", _extends({\n    parentName: \"p\"\n  }, {\n    \"className\": \"language-text\"\n  }), `isGraphemeLike`), ` properties, instead you can count the number of items directly.`), mdx(\"h2\", {\n    \"id\": \"support-for-other-languages\"\n  }, `Support for other languages`), mdx(\"p\", null, `The real power is that the API doesn't just split strings on spaces or periods, but uses specific algorithms for each language. That means the API can also segment words in languages that do not have spaces, like Japanese:`), mdx(\"div\", {\n    \"className\": \"gatsby-highlight\",\n    \"data-language\": \"js\"\n  }, mdx(\"pre\", _extends({\n    parentName: \"div\"\n  }, {\n    \"className\": \"language-js\"\n  }), mdx(\"code\", _extends({\n    parentName: \"pre\"\n  }, {\n    \"className\": \"language-js\"\n  }), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token keyword\"\n  }), `const`), ` segmenter `, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token operator\"\n  }), `=`), ` `, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token keyword\"\n  }), `new`), ` `, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token class-name\"\n  }), `Intl`, mdx(\"span\", _extends({\n    parentName: \"span\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `.`), `Segmenter`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `(`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token string\"\n  }), `'ja'`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `,`), ` `, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `{`), ` `, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token literal-property property\"\n  }), `granularity`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token operator\"\n  }), `:`), ` `, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token string\"\n  }), `'word'`), ` `, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `}`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `)`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `;`), `\n`, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token keyword\"\n  }), `const`), ` segments `, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token operator\"\n  }), `=`), ` segmenter`, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `.`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token function\"\n  }), `segment`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `(`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token string\"\n  }), `'これは日本語のテキストです'`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `)`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `;`), `\n\nArray`, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `.`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token function\"\n  }), `from`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `(`), `segments`, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `)`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `.`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token function\"\n  }), `map`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `(`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `(`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token parameter\"\n  }), `segment`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `)`), ` `, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token operator\"\n  }), `=>`), ` segment`, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `.`), `segment`, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `)`), mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token punctuation\"\n  }), `;`), `\n`, mdx(\"span\", _extends({\n    parentName: \"code\"\n  }, {\n    \"className\": \"token comment\"\n  }), `// ['これ', 'は', '日本語', 'の', 'テキスト', 'です']`)))), mdx(\"p\", null, `Now to be fair, I can not read Japanese so I have no idea if this segmentation is correct, but it sure looks convincing.`), mdx(\"p\", null, `Compared to string splitting or regexes, this is a lot more accurate and handles non-latin characters and punctuation correctly, needs less code and doesn't need additional checks and tests. Browsers ship with understanding of a large number of languages, so you don't have to worry about that either.`), mdx(\"h2\", {\n    \"id\": \"dig-into-the-intl-api\"\n  }, `Dig into the Intl API`), mdx(\"p\", null, `The Intl API has an incredible number of useful tools for creating natural language constructs like lists, dates, money, numbers and more. By and large it's easy to work with, but it uses a lot of very precise terminology that you might not be familiar with (for example, \"grapheme\" instead of \"character\", because the latter is too ambiguous), so it can be a bit daunting to get started with.`), mdx(\"p\", null, `For a full overview of the API, check out our article on `, mdx(\"a\", _extends({\n    parentName: \"p\"\n  }, {\n    \"href\": \"/blog/the-intl-api-the-best-browser-api-youre-not-using/\"\n  }), `the Intl API`), `. It covers the entire API with lots of demos and explanations of the different parts of the API and how to use them.`), mdx(\"p\", null, `As for Polypane, you can bet the next version will have text measurement option that works across languages and also tells you the number of sentences in any text you've selected.`));\n}\n;\nMDXContent.isMDXComponent = true;","timeToRead":3,"tableOfContents":{"items":[{"url":"#how-we-use-the-segmenter-api-in-polypane","title":"How we use the segmenter API in Polypane"},{"url":"#setting-up-the-segmenter-api","title":"Setting up the segmenter API"},{"url":"#support-for-other-languages","title":"Support for other languages"},{"url":"#dig-into-the-intl-api","title":"Dig into the Intl API"}]},"excerpt":"The  Intl  API in browsers has a ton of functionality around editing and formatting of text and numbers. While filling in the  State of HTML  survey I came…","frontmatter":{"title":"Using the Intl segmenter API","cover":"/blogs/segmenter/cover.svg","date":"2023-10-03","updated":null},"fields":{"slug":"/blog/using-the-intl-segmenter-api/","date":"2023-10-02T22:00:00.000Z","ogFileName":"using-the-intl-segmenter-api"}}},"pageContext":{"slug":"/blog/using-the-intl-segmenter-api/"}},"staticQueryHashes":["4164364741","425175329"]}