3 d" @sdZddlZyddlmZddlmZWn,ek rTddZddZd d ZYnXej d Z Gd d d Z ddZ ddZ e fddZe fddZddZddZdS)z A port of the Gale-Church Aligner. Gale & Church (1993), A Program for Aligning Sentences in Bilingual Corpora. https://aclweb.org/anthology/J93-1004.pdf N)logsf)normcCst|}ddd|}|tj| |d|d|d|d|d|d|d|d |d|d }|d kr|Sd|SdS)zComplementary error function.g?gś??g5?g`yg?gƸ?gꪂI?g#v?g9)?gS?gޅ1O?gv(?gg@NgꪂIǿg9)gޅ1O)absmathexp)xztrr :/tmp/pip-build-v9q4h5k9/nltk/nltk/translate/gale_church.pyerfccs(4rcCsddt|tjdS)u>Return the area under the normal distribution from M{-∞..x}.rg?)rrsqrt)rr r r norm_cdf@src Cs0ytjdt|Stk r*tdSXdS)Nrz-inf)rlogr ValueErrorfloat)rr r r norm_logsfDsrrc @s0eZdZd dd dd dd dddddiZdZdZd S)LanguageIndependentrrgׁsF?g{Gz?rgbX9ȶ?gI +?g333333@N)rr)rr)rr)rr)rr)rr)__name__ __module__ __qualname__PRIORSAVERAGE_CHARACTERSVARIANCE_CHARACTERSr r r r rNsrc Csg}t|t|f}x|dkrtdd|Dry||\}}Wn.tk rn|dd|ddf}wYnXxHt|D]<}x6t|D]*}|j|d|d|d|dfqWqzW|d||d|f}qW|dddS)a Traverse the alignment cost from the tracebacks and retrieves appropriate sentence pairs. :param backlinks: A dictionary where the key is the alignment points and value is the cost (referencing the LanguageIndependent.PRIORS) :type backlinks: dict :param source_sents_lens: A list of target sentences' lengths :type source_sents_lens: list(int) :param target_sents_lens: A list of target sentences' lengths :type target_sents_lens: list(int) rcss|]}|dkVqdS)rNr ).0pr r r nsztrace..rN)rr)lenall TypeErrorrangeappend) backlinkssource_sents_lenstarget_sents_lenslinkspositionsr ijr r r trace`s .r.c stfddt|dD}tfddt|dD}y4|||jd}||j|tj||j} Wntk rtdSXtt t | tj |j | S)aPReturns the log probability of the two sentences C{source_sents[i]}, C{target_sents[j]} being aligned with a specific C{alignment}. @param i: The offset of the source sentence. @param j: The offset of the target sentence. @param source_sents: The list of source sentence lengths. @param target_sents: The list of target sentence lengths. @param alignment: The alignment type, a tuple of two integers. @param params: The sentence alignment parameters. @returns: The log probability of a specific alignment between the two sentences, given the parameters. c3s|]}|dVqdS)rNr )roffset)r, source_sentsr r rsz!align_log_prob..rc3s|]}|dVqdS)rNr )rr/)r- target_sentsr r rsrrz-inf) sumr$rrrrZeroDivisionErrorrLOG2rrrr) r,r-r0r1 alignmentparamsZl_sZl_tmdeltar )r,r-r0r1r align_log_prob|s   r9c Cst|jj}gg}i}xtt|dD]}xtt|dD]}td}d} xj|D]b} d| d} || d} | t| ksV| dkrqV|| | t||||| |} | |krV| }| } qVW|tdkrd}| |||f<|dj|q@Wt|dkr|jd|jgq*Wt |||S)aReturn the sentence alignment of two text blocks (usually paragraphs). >>> align_blocks([5,5,5], [7,7,7]) [(0, 0), (1, 1), (2, 2)] >>> align_blocks([10,5,5], [12,20]) [(0, 0), (1, 1), (2, 1)] >>> align_blocks([12,20], [10,5,5]) [(0, 0), (1, 1), (1, 2)] >>> align_blocks([10,2,10,10,2,10], [12,3,20,3,12]) [(0, 0), (1, 1), (2, 2), (3, 2), (4, 3), (5, 4)] @param source_sents_lens: The list of source sentence lengths. @param target_sents_lens: The list of target sentence lengths. @param params: the sentence alignment parameters. @return: The sentence alignments, a list of index pairs. rinfNrrr r ) listrkeysr$r!rr9r%popr.)r'r(r6Zalignment_typesDr&r,r-Zmin_distZ min_alignaZprev_iZprev_jrr r r align_blockss2       r@cs0t|t|krtdfddt||DS)aCreates the sentence alignment of two texts. Texts can consist of several blocks. Block boundaries cannot be crossed by sentence alignment links. Each block consists of a list that contains the lengths (in characters) of the sentences in this block. @param source_blocks: The list of blocks in the source text. @param target_blocks: The list of blocks in the target text. @param params: the sentence alignment parameters. @returns: A list of sentence alignment lists z>Source and target texts do not have the same number of blocks.csg|]\}}t||qSr )r@)rZ source_blockZ target_block)r6r r szalign_texts..)r!rzip)Z source_blocksZ target_blocksr6r )r6r align_textss  rCc#s&fdd}x|jVqWdS)zSplits an iterator C{it} at values of C{split_value}. Each instance of C{split_value} is swallowed. The iterator produces subiterators which need to be consumed fully before the next subiterator can be used. c3s$|}x|kr|Vj}qWdS)N)next)firstv)it split_valuer r _chunk_iterators z!split_at.._chunk_iteratorN)rD)rGrHrIr )rGrHr split_atsrJcsfddt||DS)zParses a stream of tokens and splits it into sentences (using C{soft_delimiter} tokens) and blocks (using C{hard_delimiter} tokens) for use with the L{align_texts} function. cs g|]}ddt|DqS)cSsg|]}tdd|DqS)css|]}t|VqdS)N)r!)rtokenr r r rsz;parse_token_stream....)r2)rZ sentence_itr r r rAsz1parse_token_stream...)rJ)rZblock_it)soft_delimiterr r rAsz&parse_token_stream..)rJ)streamrLZhard_delimiterr )rLr parse_token_streams rN)__doc__rrrrZ scipy.stats ImportErrorrrrr4rr.r9r@rCrJrNr r r r s  '  6