Package: wnpp Severity: wishlist Subject: ITP: r-cran-tokenizers -- GNU R fast, consistent tokenization of natural language text Package: wnpp Owner: Andreas Tille <ti...@debian.org> Severity: wishlist
* Package name : r-cran-tokenizers Version : 0.2.1 Upstream Author : Lincoln Mullen, * URL : https://cran.r-project.org/package=tokenizers * License : MIT Programming Lang: GNU R Description : GNU R fast, consistent tokenization of natural language text Convert natural language text into tokens. Includes tokenizers for shingled n-grams, skip n-grams, words, word stems, sentences, paragraphs, characters, shingled characters, lines, tweets, Penn Treebank, regular expressions, as well as functions for counting characters, words, and sentences, and a function for splitting longer texts into separate documents, each with the same number of words. The tokenizers have a consistent interface, and the package is built on the 'stringi' and 'Rcpp' packages for fast yet correct tokenization in 'UTF-8'. Remark: This package is maintained by Debian R Packages Maintainers at https://salsa.debian.org/r-pkg-team/r-cran-tokenizers
