3 ­d5 ã@sddZddlmZmZddlmZmZmZddlm Z ddl m Z Gdd„deƒZ Gdd „d e ƒZ d S) z Tokenizer Interface é)ÚABCÚabstractmethod)ÚIteratorÚListÚTuple)Ú overridden)Ústring_span_tokenizec@s„eZdZdZeeeedœdd„ƒZeee e e fdœdd„Z eeeeedœdd „Z eeeee e e fdœd d „Z d S) Ú TokenizerIz† A processing interface for tokenizing a string. Subclasses must define ``tokenize()`` or ``tokenize_sents()`` (or both). )ÚsÚreturncCst|jƒr|j|gƒdSdS)zL Return a tokenized copy of *s*. :rtype: List[str] rN)rÚtokenize_sents)Úselfr ©rú1/tmp/pip-build-v9q4h5k9/nltk/nltk/tokenize/api.pyÚtokenizes zTokenizerI.tokenizecCs tƒ‚dS)z» Identify the tokens using integer offsets ``(start_i, end_i)``, where ``s[start_i:end_i]`` is the corresponding token. :rtype: Iterator[Tuple[int, int]] N)ÚNotImplementedError)r r rrrÚ span_tokenize$szTokenizerI.span_tokenize)Ústringsr cs‡fdd„|DƒS)z« Apply ``self.tokenize()`` to each element of ``strings``. I.e.: return [self.tokenize(s) for s in strings] :rtype: List[List[str]] csg|]}ˆj|ƒ‘qSr)r)Ú.0r )r rrú 5sz-TokenizerI.tokenize_sents..r)r rr)r rr -szTokenizerI.tokenize_sentsccs"x|D]}t|j|ƒƒVqWdS)z» Apply ``self.span_tokenize()`` to each element of ``strings``. I.e.: return [self.span_tokenize(s) for s in strings] :yield: List[Tuple[int, int]] N)Úlistr)r rr rrrÚspan_tokenize_sents7s zTokenizerI.span_tokenize_sentsN)Ú__name__Ú __module__Ú __qualname__Ú__doc__rÚstrrrrrÚintrr rrrrrr s   r c@s0eZdZdZeedd„ƒƒZdd„Zdd„ZdS) ÚStringTokenizerzxA tokenizer that divides a string into substrings by splitting on the specified string (defined in subclasses). cCst‚dS)N)r)r rrrÚ_stringJszStringTokenizer._stringcCs |j|jƒS)N)Úsplitr)r r rrrrOszStringTokenizer.tokenizeccst||jƒEdHdS)N)rr)r r rrrrRszStringTokenizer.span_tokenizeN) rrrrÚpropertyrrrrrrrrrEs rN)rÚabcrrZtypingrrrZnltk.internalsrZnltk.tokenize.utilrr rrrrrÚ s   1