Skip to main content
Version: Next

Module: text/BasicTokenizer

Classes​

Type aliases​

BasicTokenizerConfig​

Ƭ BasicTokenizerConfig: Object

Copyright (c) Meta Platforms, Inc. and affiliates.

This source code is licensed under the MIT license found in the LICENSE file in the root directory of this source tree.

format

Type declaration​

NameType
lowercase?boolean
neverSplit?string[]