3 d@ @sdZddlZddlmZmZddlmZddZej Z ddZ dZ ydd l mZWnek rnd d ZYnXdZdZdZGdddedZGdddeZGdddeZGdddeZGdddZdS)z Provides scoring functions for a number of association measures through a generic, abstract implementation in ``NgramAssocMeasures``, and n-specific ``BigramAssocMeasures`` and ``TrigramAssocMeasures``. N)ABCMetaabstractmethod)reducecCs tj|S)N)_mathlog2)xr8/tmp/pip-build-v9q4h5k9/nltk/nltk/metrics/association.pysr cCstdd|S)NcSs||S)Nr)ryrrr r sz..)r)srrr r sg#B ;) fisher_exactcOstdS)N)NotImplementedError)_args_kwargsrrr r sr c@seZdZdZdZeeddZeeddZe ddZ ed d Z e d d Z e d dZ eddZe ddZe ddZe ddZe ddZdS)NgramAssocMeasuresa An abstract class defining a collection of generic association measures. Each public method returns a score, taking the following arguments:: score_fn(count_of_ngram, (count_of_n-1gram_1, ..., count_of_n-1gram_j), (count_of_n-2gram_1, ..., count_of_n-2gram_k), ..., (count_of_1gram_1, ..., count_of_1gram_n), count_of_total_words) See ``BigramAssocMeasures`` and ``TrigramAssocMeasures`` Inheriting classes should define a property _n, and a method _contingency which calculates contingency values from marginals in order for all association measures defined here to be usable. rcGs tddS)z>Calculates values of a contingency table from marginal values.z?The contingency table is not availablein the general ngram caseN)r) marginalsrrr _contingencyBszNgramAssocMeasures._contingencycGs tddS)zACalculates values of contingency table marginals from its values.z?The contingency table is not availablein the general ngram caseN)r) contingencyrrr _marginalsJszNgramAssocMeasures._marginalsc#s`t}ddtjD}x>ttD].tfdd|D|jdVq*WdS)z3Calculates expected values for a contingency table.cSsg|] }d|>qS)rr).0irrr Vsz7NgramAssocMeasures._expected_values..c3s2|]*tfddtdjDVqdS)c3s&|]}|@@kr|VqdS)Nr)rr)contrjrr ]sz@NgramAssocMeasures._expected_values...rN)sumrange_n)r)clsrr)rr r]sz6NgramAssocMeasures._expected_values..rN)rrr len_product)r!rn_allbitsr)r!rrr _expected_valuesRsz#NgramAssocMeasures._expected_valuescGs|t|tS)z Scores ngrams by their frequency)NGRAMTOTAL)rrrr raw_freqcszNgramAssocMeasures.raw_freqcGs6|tt|t|t|jd|ttdS)zScores ngrams using Student's t test with independence hypothesis for unigrams, as in Manning and Schutze 5.3.1. rg?)r'r#UNIGRAMSr(r _SMALL)r!rrrr student_thszNgramAssocMeasures.student_tcGs,|j|}|j|}tddt||DS)zZScores ngrams using Pearson's chi-square as in Manning and Schutze 5.3.3. css&|]\}}||d|tVqdS)rN)r+)robsexprrr rysz,NgramAssocMeasures.chi_sq..)rr&rzip)r!rrZexpsrrr chi_sqrs  zNgramAssocMeasures.chi_sqcOs |t|jddt|tS)zScores ngrams using a variant of mutual information. The keyword argument power sets an exponent (default 3) for the numerator. No logarithm of the result is calculated. power)r'getr#r*)rkwargsrrr mi_like{szNgramAssocMeasures.mi_likecGs.t|t|t|jdtt|tS)z^Scores ngrams by pointwise mutual information, as in Manning and Schutze 5.4. r)_log2r'r(r r#r*)r!rrrr pmiszNgramAssocMeasures.pmicGs,|j|}dtddt||j|DS)zFScores ngrams using likelihood ratios as in Manning and Schutze 5.3.4.rcss*|]"\}}|t||ttVqdS)N)_lnr+)rr-r.rrr rsz6NgramAssocMeasures.likelihood_ratio..)rrr/r&)r!rrrrr likelihood_ratios z#NgramAssocMeasures.likelihood_ratiocGs:t|t|t|jd}|tt|t|dS)z1Scores ngrams using the Poisson-Stirling measure.r)r#r*r(r r'r6)r!rr.rrr poisson_stirlingsz#NgramAssocMeasures.poisson_stirlingcGs"|j|}|dt|ddS)z&Scores ngrams using the Jaccard index.rNr)rr)r!rrrrr jaccards zNgramAssocMeasures.jaccardN)__name__ __module__ __qualname____doc__r staticmethodrrr classmethodr&r)r,r0r5r7r9r:r<rrrr r-s   r) metaclassc@sheZdZdZdZeddZeddZeddZe d d Z e d d Z e d dZ eddZ dS)BigramAssocMeasuresa A collection of bigram association measures. Each association measure is provided as a function with three arguments:: bigram_score_fn(n_ii, (n_ix, n_xi), n_xx) The arguments constitute the marginals of a contingency table, counting the occurrences of particular events in a corpus. The letter i in the suffix refers to the appearance of the word in question, while x indicates the appearance of any word. Thus, for example: - n_ii counts ``(w1, w2)``, i.e. the bigram being scored - n_ix counts ``(w1, *)`` - n_xi counts ``(*, w2)`` - n_xx counts ``(*, *)``, i.e. any bigram This may be shown with respect to a contingency table:: w1 ~w1 ------ ------ w2 | n_ii | n_oi | = n_xi ------ ------ ~w2 | n_io | n_oo | ------ ------ = n_ix TOTAL = n_xx rcCs0|\}}||}||}|||||||fS)zECalculates values of a bigram contingency table from marginal values.r)n_ii n_ix_xi_tuplen_xxn_ixn_xin_oin_iorrr rsz BigramAssocMeasures._contingencycCs"|||||f||||fS)zACalculates values of contingency table marginals from its values.r)rErJrKn_oorrr rszBigramAssocMeasures._marginalsccsLt|}x>tdD]2}||||dA||||dA|VqWdS)z3Calculates expected values for a contingency table.rrN)rr)rrGrrrr r&sz$BigramAssocMeasures._expected_valuescGsF|j|\}}}}||||d||||||||S)zdScores bigrams using phi-square, the square of the Pearson correlation coefficient. r)r)r!rrErKrJrLrrr phi_sqszBigramAssocMeasures.phi_sqcCs|\}}||j|||f|S)zScores bigrams using chi-square, i.e. phi-sq multiplied by the number of bigrams, as in Manning and Schutze 5.3.3. )rN)r!rErFrGrHrIrrr r0szBigramAssocMeasures.chi_sqcGs2|j|\}}}}t||g||ggdd\}}|S)zScores bigrams using Fisher's Exact Test (Pedersen 1996). Less sensitive to small counts than PMI or Chi Sq, but also more expensive to compute. Requires scipy. Zless) alternative)rr )r!rrErKrJrLZoddsZpvaluerrr fisherszBigramAssocMeasures.fishercCs|\}}d|||S)z(Scores bigrams using Dice's coefficient.rr)rErFrGrHrIrrr diceszBigramAssocMeasures.diceN)r=r>r?r@r rArrr&rBrNr0rPrQrrrr rDs     rDc@s,eZdZdZdZeddZeddZdS)TrigramAssocMeasuresa A collection of trigram association measures. Each association measure is provided as a function with four arguments:: trigram_score_fn(n_iii, (n_iix, n_ixi, n_xii), (n_ixx, n_xix, n_xxi), n_xxx) The arguments constitute the marginals of a contingency table, counting the occurrences of particular events in a corpus. The letter i in the suffix refers to the appearance of the word in question, while x indicates the appearance of any word. Thus, for example: - n_iii counts ``(w1, w2, w3)``, i.e. the trigram being scored - n_ixx counts ``(w1, *, *)`` - n_xxx counts ``(*, *, *)``, i.e. any trigram r2cCs|\}}}|\}}} ||} ||} ||} | || | } ||| | }||| | }||| | | | ||}|| | | | |||fS)zCalculates values of a trigram contingency table (or cube) from marginal values. >>> TrigramAssocMeasures._contingency(1, (1, 1, 1), (1, 73, 1), 2000) (1, 0, 0, 0, 0, 72, 0, 1927) r)n_iiiZ n_iix_tupleZ n_ixx_tupleZn_xxxZn_iixZn_ixiZn_xiiZn_ixxZn_xixZn_xxin_oiin_ioin_iion_ooin_oion_ioon_ooorrr rs   z!TrigramAssocMeasures._contingencyc Gs`|\}}}}}}}}|||||||f||||||||||||ft|fS)zCalculates values of contingency table marginals from its values. >>> TrigramAssocMeasures._marginals(1, 0, 0, 0, 0, 72, 0, 1927) (1, (1, 1, 1), (1, 73, 1), 2000) )r) rrSrTrUrWrVrXrYrZrrr r&szTrigramAssocMeasures._marginalsN)r=r>r?r@r rArrrrrr rRs rRc@s,eZdZdZdZeddZeddZdS)QuadgramAssocMeasuresaF A collection of quadgram association measures. Each association measure is provided as a function with five arguments:: trigram_score_fn(n_iiii, (n_iiix, n_iixi, n_ixii, n_xiii), (n_iixx, n_ixix, n_ixxi, n_xixi, n_xxii, n_xiix), (n_ixxx, n_xixx, n_xxix, n_xxxi), n_all) The arguments constitute the marginals of a contingency table, counting the occurrences of particular events in a corpus. The letter i in the suffix refers to the appearance of the word in question, while x indicates the appearance of any word. Thus, for example: - n_iiii counts ``(w1, w2, w3, w4)``, i.e. the quadgram being scored - n_ixxi counts ``(w1, *, *, w4)`` - n_xxxx counts ``(*, *, *, *)``, i.e. any quadgram rMc"Cs|\}}}}|\} } } } } }|\}}}}||}||}||}| |||}| |||}| |||}||||||||}||}||||}| |||}||||||||}| |||}||||||||}||||||||} |||||||||||||||| }!||||||||||||||| |!fS)zXCalculates values of a quadgram contingency table from marginal values. r)"n_iiiiZ n_iiix_tupleZ n_iixx_tupleZ n_ixxx_tupleZn_xxxxn_iiixn_iixin_ixiin_xiiin_iixxn_ixixn_ixxin_xixin_xxiin_xiixn_ixxxn_xixxn_xxixn_xxxin_oiiin_ioiin_iioin_ooiin_oioin_iooin_oooin_iiion_oiion_ioion_ooion_iioon_oioon_iooon_oooorrr rPsD      @z"QuadgramAssocMeasures._contingencyc GsV|\}}}}}}}}} } } } } }}}|| }||}||}||}||| | }||| | }||||}||||}||||}||| | }|||| || | |}|||| || | |}|||| || | | }||||||||}t|}|||||f||||||f||||f|fS)aCalculates values of contingency table marginals from its values. QuadgramAssocMeasures._marginals(1, 0, 2, 46, 552, 825, 2577, 34967, 1, 0, 2, 48, 7250, 9031, 28585, 356653) (1, (2, 553, 3, 1), (7804, 6, 3132, 1378, 49, 2), (38970, 17660, 100, 38970), 440540) )r) rr\rkrlrnrmrorprqrrrsrtrurvrwrxryr]r^r_r`rarbrcrdrerfrgrhrirjr$rrr rs*$      z QuadgramAssocMeasures._marginalsN)r=r>r?r@r rArrrrrr r[9s .res)r@r=)rrrr)rrr rsz(ContingencyMeasures._make_contingency_fnN)r=r>r?r@rrArrrrr rzs rzr;)r@mathrabcrr functoolsrr6logr8r#r+Z scipy.statsr ImportErrorr'r*r(rrDrRr[rzrrrr  s( wY<