3 d@s,dZddlZddlmZGdddeZdS)a The tok-tok tokenizer is a simple, general tokenizer, where the input has one sentence per line; thus only final period is tokenized. Tok-tok has been tested on, and gives reasonably good results for English, Persian, Russian, Czech, French, German, Vietnamese, Tajik, and a few others. The input should be in UTF-8 encoding. Reference: Jon Dehdari. 2014. A Neurophysiologically-Inspired Statistical Language Model (Doctoral dissertation). Columbus, OH, USA: The Ohio State University. N) TokenizerIc@seZdZdZejddfZejddfZejddfZejddfZ ejdd fZ ejd d fZ ejd d fZ ejddfZ ejddfZejddfZejddfZejddfZejddfZejddfZejddfZejddfZedZedZedZejdedd fZejdedd fZejdedd fZejd!d"fZejd#d$fZejd%d&fZejd'd&fZ ejd(d)fZ!ejd*d+fZ"ejd,dfZ#eeeeee e e e eeee eeeeee eeeee#gZ$d1d.d/Z%d0S)2ToktokTokenizeru This is a Python port of the tok-tok.pl from https://github.com/jonsafari/tok-tok/blob/master/tok-tok.pl >>> toktok = ToktokTokenizer() >>> text = u'Is 9.5 or 525,600 my favorite number?' >>> print(toktok.tokenize(text, return_str=True)) Is 9.5 or 525,600 my favorite number ? >>> text = u'The https://github.com/jonsafari/tok-tok/blob/master/tok-tok.pl is a website with/and/or slashes and sort of weird : things' >>> print(toktok.tokenize(text, return_str=True)) The https://github.com/jonsafari/tok-tok/blob/master/tok-tok.pl is a website with/and/or slashes and sort of weird : things >>> text = u'¡This, is a sentence with weird» symbols… appearing everywhere¿' >>> expected = u'¡ This , is a sentence with weird » symbols … appearing everywhere ¿' >>> assert toktok.tokenize(text, return_str=True) == expected >>> toktok.tokenize(text) == [u'¡', u'This', u',', u'is', u'a', u'sentence', u'with', u'weird', u'»', u'symbols', u'…', u'appearing', u'everywhere', u'¿'] True   u1([،;؛¿!"\])}»›”؟¡%٪°±©®।॥…])z \1 u([({\[“‘„‚«‹「『])u ([–—])z& z&  z z\|z | u(?s