3
d @ sZ d dl mZ d dlmZ d dlmZmZ ejZeedddddZ de _
d d
Z d d Z d
S ) )partial)chain)
everygrams pad_sequenceTzz)Zpad_leftZleft_pad_symbolZ pad_rightZright_pad_symbolzPads both ends of a sentence to length specified by ngram order.
Following convention pads the start of sentence pads its end.
c C s t tt|| d| dS )zpHelper with some useful defaults.
Applies pad_both_ends to sentence and follows it up with everygrams.
)n)max_len)r list
pad_both_ends)orderZsentence r 5/tmp/pip-build-v9q4h5k9/nltk/nltk/lm/preprocessing.pypadded_everygrams s r
c s. t t d fdd|D tt|fS )a Default preprocessing for a sequence of sentences.
Creates two iterators:
- sentences padded and turned into sequences of `nltk.util.everygrams`
- sentences padded as above and chained together for a flat stream of words
:param order: Largest ngram length produced by `everygrams`.
:param text: Text to iterate over. Expected to be an iterable of sentences.
:type text: Iterable[Iterable[str]]
:return: iterator over text as ngrams, iterator over text as vocabulary data
)r c 3 s" | ]}t t| d V qdS ))r N)r r ).0sent)r
padding_fnr r 1 s z,padded_everygram_pipeline..)r r flattenmap)r
textr )r
r r padded_everygram_pipeline" s
r N)
functoolsr itertoolsr Z nltk.utilr r
from_iterabler r __doc__r
r r r r r s