3 dp @s2dZddlZddlZddlmZmZmZddlmZddl m Z ddl m Z ddl mZddlmZdd lmZmZdd lmZdd lmZdd lmZdd lmZmZeddddddddgZGdddZGdddZGdddZ GdddZ!Gddde!Z"d d!Z#e$d"kr e#dddddgZ%dS)#a  This module brings together a variety of NLTK functionality for text analysis, and provides simple, interactive interfaces. Functionality includes: concordancing, collocation discovery, regular expression search over tokenized strings, and distributional similarity. N)Counter defaultdict namedtuple)reduce)log)BigramCollocationFinder)MLE)padded_everygram_pipeline)BigramAssocMeasures f_measure)ConditionalFreqDist)FreqDist) sent_tokenize)LazyConcatenation tokenwrapConcordanceLineleftqueryrightoffset left_print right_printlinec@sTeZdZdZeddZddddfddZd d Zd d ZdddZ dddZ dS) ContextIndexa A bidirectional index between words and their 'contexts' in a text. The context of a word is usually defined to be the words that occur in a fixed window around the word; but other definitions may also be used by providing a custom context function. cCsH|dkr||djnd}|t|dkr<||djnd}||fS)z;One left token and one right token, normalized to lowercaserz*START*z*END*)lowerlen)tokensirrr)/tmp/pip-build-v9q4h5k9/nltk/nltk/text.py_default_context.s$zContextIndex._default_contextNcCs|S)Nr)xrrr 5szContextIndex.csv|__|r|_nj_r6fddDtfddtD_tfddtD_dS)Ncsg|]}|r|qSrr).0t)filterrr =sz)ContextIndex.__init__..c3s(|] \}}j|j|fVqdS)N)_key _context_func)r$rw)selfrrr ?sz(ContextIndex.__init__..c3s(|] \}}j|j|fVqdS)N)r)r()r$rr*)r+rrr r,Bs)r(_tokensr)r!CFD enumerate_word_to_contexts_context_to_words)r+rZ context_funcr&keyr)r&r+rr __init__5szContextIndex.__init__cCs|jS)zw :rtype: list(str) :return: The document that this context index was created from. )r-)r+rrr rEszContextIndex.tokenscCsJ|j|}t|j|}i}x(|jjD]\}}t|t|||<q(W|S)z Return a dictionary mapping from words to 'similarity scores,' indicating how often these two words occur in the same context. )r(setr0itemsr )r+wordZ word_contextsscoresr*Z w_contextsrrr word_similarity_dictMs  z!ContextIndex.word_similarity_dictcCs~tt}x\|j|j|D]H}xB|j|D]4}||kr*|||j|||j||7<q*WqWt||jddd|S)NT)r2reverse)rintr0r(r1sortedget)r+r6nr7cr*rrr similar_words\s(zContextIndex.similar_wordsFcsfddDfddDfddttD}ttj|rf|rftddjn&sptStfddD}|Sd S) a Find contexts where the specified words can all appear; and return a frequency distribution mapping each context to the number of times that context was used. :param words: The words used to seed the similarity search :type words: str :param fail_on_unknown: If true, then raise a value error if any of the given words do not occur at all in the index. csg|]}j|qSr)r()r$r*)r+rr r'qsz0ContextIndex.common_contexts..csg|]}tj|qSr)r4r0)r$r*)r+rr r'rscsg|]}|s|qSrr)r$r)contextswordsrr r'ssz%The following word(s) were not found: c3s*|]"}j|D]}|kr|VqqdS)N)r0)r$r*r?)commonr+rr r,|sz/ContextIndex.common_contexts..N)rangerrr4 intersection ValueErrorjoinr )r+rBZfail_on_unknownemptyfdr)rDrAr+rBr common_contextsfs  zContextIndex.common_contexts)r9)F) __name__ __module__ __qualname____doc__ staticmethodr!r3rr8r@rKrrrr r&s  rc@sLeZdZdZddfddZddZdd Zd d Zdd dZdddZ dS)ConcordanceIndexzs An index that can be used to look up the offset locations at which a given word occurs in a document. cCs|S)Nr)r"rrr r#szConcordanceIndex.cCsJ||_||_tt|_x.t|D]"\}}|j|}|j|j|q WdS)a Construct a new concordance index. :param tokens: The document (list of tokens) that this concordance index was created from. This list can be used to access the context of a given word occurrence. :param key: A function that maps each token to a normalized version that will be used as a key in the index. E.g., if you use ``key=lambda s:s.lower()``, then the index will be case-insensitive. N)r-r(rlist_offsetsr/append)r+rr2indexr6rrr r3s   zConcordanceIndex.__init__cCs|jS)z{ :rtype: list(str) :return: The document that this concordance index was created from. )r-)r+rrr rszConcordanceIndex.tokenscCs|j|}|j|S)z :rtype: list(int) :return: A list of the offset positions at which the given word occurs. If a key function was specified for the index, then given word's key will be looked up. )r(rS)r+r6rrr offsetss zConcordanceIndex.offsetscCsdt|jt|jfS)Nz+)rr-rS)r+rrr __repr__szConcordanceIndex.__repr__Pc sHt|tr|}n|g}|tdj|dd}|d}g}|j|d}xBt|ddD].\}fdd|j|D}t|j|}q\W|rDx|D]dj|jt|} |jt d|} |jt||} dj| | d} dj| d|} dj| | | g}t | | | | | |}|j |qW|S) z Find all concordance lines given the query word. Provided with a list of words, these will be found as a phrase. rCrrNcsh|]}|dqS)rr)r$r)rrr sz4ConcordanceIndex.find_concordance..) isinstancerRrrHrVr/r<rFr-maxrrT)r+r6widthphraseZ half_widthcontextconcordance_listrVZ word_offsetsZ query_wordZ left_contextZ right_contextrrZ line_printconcordance_liner)rr find_concordances8  z!ConcordanceIndex.find_concordancecCsp|j||d}|stdnPt|t|}td|dt|dx&t|d|D]\}}t|jqVWdS)a Print concordance lines given the query word. :param word: The target word or phrase (a list of strings) :type word: str or list :param lines: The number of lines to display (default=25) :type lines: int :param width: The width of each line, in characters (default=80) :type width: int :param save: The option to save the concordance. :type save: bool )r^z no matchesz Displaying z of z matches:N)rcprintminrr/r)r+r6r^linesrarrbrrr print_concordances  z"ConcordanceIndex.print_concordanceN)rX)rXrd) rLrMrNrOr3rrVrWrcrhrrrr rQs  ,rQc@s eZdZdZddZddZdS) TokenSearchera A class that makes it easier to use regular expressions to search over tokenized strings. The tokenized string is converted to a string where tokens are marked with angle brackets -- e.g., ``''``. The regular expression passed to the ``findall()`` method is modified to treat angle brackets as non-capturing parentheses, in addition to matching the token boundaries; and to have ``'.'`` not match the angle brackets. cCsdjdd|D|_dS)Ncss|]}d|dVqdS)<>Nr)r$r*rrr r,sz)TokenSearcher.__init__..)rH_raw)r+rrrr r3szTokenSearcher.__init__cCstjdd|}tjdd|}tjdd|}tjdd|}tj||j}x*|D]"}|jd rL|jdrLtd qLWd d |D}|S) a" Find instances of the regular expression in the text. The text is a list of tokens, and a regexp pattern to match a single token must be surrounded by angle brackets. E.g. >>> from nltk.text import TokenSearcher >>> print('hack'); from nltk.book import text1, text5, text9 hack... >>> text5.findall("<.*><.*>") you rule bro; telling you bro; u twizted bro >>> text1.findall("(<.*>)") monied; nervous; dangerous; white; white; white; pious; queer; good; mature; white; Cape; great; wise; wise; butterless; white; fiendish; pale; furious; better; certain; complete; dismasted; younger; brave; brave; brave; brave >>> text9.findall("{3,}") thread through those; the thought that; that the thing; the thing that; that that thing; through these than through; them that the; through the thick; them that they; thought that the :param regexp: A regular expression :type regexp: str z\srjrkz(?:<(?:rlz)>)z (?]z$Bad regexp for TokenSearcher.findallcSsg|]}|ddjdqS)rz><)split)r$hrrr r'1sz)TokenSearcher.findall..)resubfindallrm startswithendswithrG)r+regexphitsrprrr rs s  zTokenSearcher.findallN)rLrMrNrOr3rsrrrr ris ric@seZdZdZdZd6ddZddZdd Zd7d d Zd8ddZ d9ddZ d:ddZ ddZ ddZ ddZd;ddZdd'd(Zd)d*Zd+d,Zd-d.Zejd/Zd0d1Zd2d3Zd4d5ZdS)?Texta A wrapper around a sequence of simple (string) tokens, which is intended to support initial exploration of texts (via the interactive console). Its methods perform a variety of analyses on the text's contexts (e.g., counting, concordancing, collocation discovery), and display the results. If you wish to write a program which makes use of these analyses, then you should bypass the ``Text`` class, and use the appropriate analysis function or class directly instead. A ``Text`` is typically initialized from a given document or corpus. E.g.: >>> import nltk.corpus >>> from nltk.text import Text >>> moby = Text(nltk.corpus.gutenberg.words('melville-moby_dick.txt')) TNcCs|jrt|}||_|r ||_ndd|ddkrb|ddjd}djdd|d|D|_n"djdd|dd Dd |_dS) zv Create a Text object. :param tokens: The source text. :type tokens: sequence of str ]Nr9rCcss|]}t|VqdS)N)str)r$tokrrr r,^sz Text.__init__..rcss|]}t|VqdS)N)rz)r$r{rrr r,`sz...) _COPY_TOKENSrRrnamerUrH)r+rr~endrrr r3Os z Text.__init__cCs |j|S)N)r)r+rrrr __getitem__fszText.__getitem__cCs t|jS)N)rr)r+rrr __len__isz Text.__len__OrdcCs.d|jkrt|jddd|_|jj|||S)a Prints a concordance for ``word`` with the specified context window. Word matching is not case-sensitive. :param word: The target word or phrase (a list of strings) :type word: str or list :param width: The width of each line, in characters (default=80) :type width: int :param lines: The number of lines to display (default=25) :type lines: int :seealso: ``ConcordanceIndex`` _concordance_indexcSs|jS)N)r)srrr r#sz"Text.concordance..)r2)__dict__rQrrrh)r+r6r^rgrrr concordanceps zText.concordancecCs4d|jkrt|jddd|_|jj||d|S)a Generate a concordance for ``word`` with the specified context window. Word matching is not case-sensitive. :param word: The target word or phrase (a list of strings) :type word: str or list :param width: The width of each line, in characters (default=80) :type width: int :param lines: The number of lines to display (default=25) :type lines: int :seealso: ``ConcordanceIndex`` rcSs|jS)N)r)rrrr r#sz'Text.concordance_list..)r2N)rrQrrrc)r+r6r^rgrrr ras zText.concordance_listr9rYcsd|jko|j|ko|j|ks||_||_ddlm}|jdtj|j|}|j d|j fddt }t |j |j||_|jS)a Return collocations derived from the text, ignoring stopwords. >>> from nltk.book import text4 >>> text4.collocation_list()[:2] [('United', 'States'), ('fellow', 'citizens')] :param num: The maximum number of collocations to return. :type num: int :param window_size: The number of tokens spanned by a collocation (default=2) :type window_size: int :rtype: list(tuple(str, str)) _collocationsr) stopwordsenglishrYcst|dkp|jkS)N)rr)r*) ignored_wordsrr r#sz'Text.collocation_list..)rZ_numZ _window_size nltk.corpusrrBrZ from_wordsrZapply_freq_filterZapply_word_filterr rRZnbestZlikelihood_ratior)r+num window_sizerfinderZbigram_measuresr)rr collocation_lists      zText.collocation_listcCs*dd|j||D}tt|dddS)a Print collocations derived from the text, ignoring stopwords. >>> from nltk.book import text4 >>> text4.collocations() # doctest: +ELLIPSIS United States; fellow citizens; four years; ... :param num: The maximum number of collocations to print. :type num: int :param window_size: The number of tokens spanned by a collocation (default=2) :type window_size: int cSsg|]\}}|d|qS)rCr)r$w1w2rrr r'sz%Text.collocations..z; ) separatorN)rrer)r+rrZcollocation_stringsrrr collocationsszText.collocationscCs |jj|S)zJ Count the number of times this word appears in the text. )rcount)r+r6rrr rsz Text.countcCs |jj|S)zQ Find the index of the first occurrence of the word in the text. )rrU)r+r6rrr rUsz Text.indexcCstdS)N)NotImplementedError)r+methodrrr readabilityszText.readabilitycsd|jkr$t|jddddd|_j|jjjkrttfddjD}dd |j |D}t t |nt d d S) a~ Distributional similarity: find other words which appear in the same contexts as the specified word; list most similar words first. :param word: The word used to seed the similarity search :type word: str :param num: The number of words to generate (default=20) :type num: int :seealso: ContextIndex.similar_words() _word_context_indexcSs|jS)N)isalpha)r"rrr r#szText.similar..cSs|jS)N)r)rrrr r#s)r&r2c3s2|]*}|D]}|kr|k r|VqqdS)Nr)r$r*r?)rAwcir6rr r,s zText.similar..cSsg|] \}}|qSrr)r$r*_rrr r'sz Text.similar..z No matchesN) rrrrrr0Z conditionsr4r most_commonrer)r+r6rrJrBr)rArr6r similars   z Text.similarcCsd|jkrt|jddd|_yJ|jj|d}|s.)r2TzNo common contexts were foundcSsg|] \}}|qSrr)r$r*rrrr r'sz(Text.common_contexts..css|]\}}|d|VqdS)rNr)r$rrrrr r,sz'Text.common_contexts..N) rrrrrKrerrrG)r+rBrrJZranked_contextserrr rKs  zText.common_contextscCsddlm}|||dS)z Produce a plot showing the distribution of the words through the text. Requires pylab to be installed. :param words: The words to be plotted :type words: list(str) :seealso: nltk.draw.dispersion_plot() r)dispersion_plotN)Z nltk.drawr)r+rBrrrr rs zText.dispersion_plotrcCs(t||\}}t|d}|j|||S)N)order)r rZfit)r+Ztokenized_sentsr>Z train_dataZ padded_sentsmodelrrr _train_default_ngram_lm*s  zText._train_default_ngram_lmd*c Csddtdj|jD|_t|dsFtdtjd|j|jdd|_ g}|d ksZt d xZt ||krx@t |j j |||d D]&\}}|d krq|d krP|j|qW|d7}q\W|rdj|dnd}|t|d|}t||S)a Print random text, generated using a trigram language model. See also `help(nltk.lm)`. :param length: The length of text to generate (default=100) :type length: int :param text_seed: Generation can be conditioned on preceding context. :type text_seed: list(str) :param random_seed: A random seed or an instance of `random.Random`. If provided, makes the random sampling part of generation reproducible. (default=42) :type random_seed: int cSsg|]}|jdqS)rC)ro)r$sentrrr r'Asz!Text.generate..rC_trigram_modelzBuilding ngram index...)filer)r>rz!The `length` must be more than 0.) text_seed random_seedzzrrjN)rrHrZ_tokenized_sentshasattrresysstderrrrAssertionErrorrr/generaterTr) r+lengthrrZgenerated_tokensidxtokenprefixZ output_strrrr r0s*  z Text.generatecGs|jj|S)zc See documentation for FreqDist.plot() :seealso: nltk.prob.FreqDist.plot() )vocabplot)r+argsrrr r^sz Text.plotcCsd|jkrt||_|jS)z. :seealso: nltk.prob.FreqDist _vocab)rr r)r+rrr res  z Text.vocabcCs@d|jkrt||_|jj|}dd|D}tt|ddS)a Find instances of the regular expression in the text. The text is a list of tokens, and a regexp pattern to match a single token must be surrounded by angle brackets. E.g. >>> print('hack'); from nltk.book import text1, text5, text9 hack... >>> text5.findall("<.*><.*>") you rule bro; telling you bro; u twizted bro >>> text1.findall("(<.*>)") monied; nervous; dangerous; white; white; white; pious; queer; good; mature; white; Cape; great; wise; wise; butterless; white; fiendish; pale; furious; better; certain; complete; dismasted; younger; brave; brave; brave; brave >>> text9.findall("{3,}") thread through those; the thought that; that the thing; the thing that; that that thing; through these than through; them that the; through the thick; them that they; thought that the :param regexp: A regular expression :type regexp: str _token_searchercSsg|]}dj|qS)rC)rH)r$rprrr r'sz Text.findall..z; N)rrirrsrer)r+rvrwrrr rsns    z Text.findallz \w+|[\.\!\?]cCs|d}x&|dkr.|jj|| r.|d8}q W|dkr@||nd}|d}x*|t|krv|jj|| rv|d7}qNW|t|kr||nd}||fS)z One left & one right token, both case-normalized. Skip over non-sentence-final punctuation. Used by the ``ContextIndex`` that is created for ``similar()`` and ``common_contexts()``. rrz*START*z*END*) _CONTEXT_REmatchr)r+rrjrrrrr _contexts   z Text._contextcCs d|jS)Nz )r~)r+rrr __str__sz Text.__str__cCs d|jS)Nz )r~)r+rrr rWsz Text.__repr__)N)rrd)rrd)r9rY)r9rY)r9)r9)r)rNr)rLrMrNrOr}r3rrrrarrrrUrrrKrrrrrrsrqcompilerrrrWrrrr rx5s0    #  "   . # rxc@s0eZdZdZddZddZddZdd Zd S) TextCollectionaVA collection of texts, which can be loaded with list of texts, or with a corpus consisting of one or more texts, and which supports counting, concordancing, collocation discovery, etc. Initialize a TextCollection as follows: >>> import nltk.corpus >>> from nltk.text import TextCollection >>> print('hack'); from nltk.book import text1, text2, text3 hack... >>> gutenberg = TextCollection(nltk.corpus.gutenberg) >>> mytexts = TextCollection([text1, text2, text3]) Iterating over a TextCollection produces all the tokens of all the texts in order. cs@tdr fddjD|_tj|ti|_dS)NrBcsg|]}j|qSr)rB)r$f)sourcerr r'sz+TextCollection.__init__..)rZfileids_textsrxr3r _idf_cache)r+rr)rr r3s  zTextCollection.__init__cCs|j|t|S)z"The frequency of the term in text.)rr)r+termtextrrr tfszTextCollection.tfcsj|jj}|dkrftfdd|jD}t|jdkrBtd|rXtt|j|nd}||j<|S)zThe number of texts in the corpus divided by the number of texts that the term appears in. If a term does not appear in the corpus, 0.0 is returned.Ncsg|]}|krdqS)Tr)r$r)rrr r'sz&TextCollection.idf..rz+IDF undefined for empty document collectiong)rr=rrrGr)r+ridfmatchesr)rr rs  zTextCollection.idfcCs|j|||j|S)N)rr)r+rrrrr tf_idfszTextCollection.tf_idfN)rLrMrNrOr3rrrrrrr rs rcCsddlm}t|jdd}t|ttd|jdttd|jdttd|jttd|jdd d d gttd |j d ttdtd|dtd|ddtd|j ddS)Nr)brownnews) categoriesz Concordance:zDistributionally similar words:z Collocations:zDispersion plot:reportZsaidZ announcedzVocabulary plot:2z Indexing:ztext[3]:rz text[3:5]:ztext.vocab()['news']:) rrrxrBrerrrrrr)rrrrr demos.    r__main__)&rOrqr collectionsrrr functoolsrmathrZnltk.collocationsrZnltk.lmrZnltk.lm.preprocessingr Z nltk.metricsr r Znltk.probabilityr r.r Z nltk.tokenizerZ nltk.utilrrrrrQrirxrrrL__all__rrrr s@        [{9/