3 d?S @sddlZddlmZddlmZddlmZddlm Z ddZGdd d Z d d Z d/ddZ ej dZd1ddZddZd3ddZddZej dejZej dZdd Zd!d"d#d$d%d&d'd(d)g d fd*d+Zd,d-Zed.kredS)4N)accuracy)map_tag) str2tuple)TreecCsFg}g}x2|D]*}|j|j}|t|7}|t|7}qWt||S)a| Score the accuracy of the chunker against the gold standard. Strip the chunk information from the gold standard and rechunk it using the chunker, then compute the accuracy score. :type chunker: ChunkParserI :param chunker: The chunker being evaluated. :type gold: tree :param gold: The chunk structures to score the chunker on. :rtype: float )parseflattentree2conlltags _accuracy)ZchunkerZgoldZ gold_tagsZ test_tagsZ gold_treeZ test_treer //tmp/pip-build-v9q4h5k9/nltk/nltk/chunk/util.pyrs   rc@seZdZdZddZddZddZdd Zd d Zd d Z d ddZ ddZ ddZ ddZ ddZddZddZddZdS)! ChunkScorea; A utility class for scoring chunk parsers. ``ChunkScore`` can evaluate a chunk parser's output, based on a number of statistics (precision, recall, f-measure, misssed chunks, incorrect chunks). It can also combine the scores from the parsing of multiple texts; this makes it significantly easier to evaluate a chunk parser that operates one sentence at a time. Texts are evaluated with the ``score`` method. The results of evaluation can be accessed via a number of accessor methods, such as ``precision`` and ``f_measure``. A typical use of the ``ChunkScore`` class is:: >>> chunkscore = ChunkScore() # doctest: +SKIP >>> for correct in correct_sentences: # doctest: +SKIP ... guess = chunkparser.parse(correct.leaves()) # doctest: +SKIP ... chunkscore.score(correct, guess) # doctest: +SKIP >>> print('F Measure:', chunkscore.f_measure()) # doctest: +SKIP F Measure: 0.823 :ivar kwargs: Keyword arguments: - max_tp_examples: The maximum number actual examples of true positives to record. This affects the ``correct`` member function: ``correct`` will not return more than this number of true positive examples. This does *not* affect any of the numerical metrics (precision, recall, or f-measure) - max_fp_examples: The maximum number actual examples of false positives to record. This affects the ``incorrect`` member function and the ``guessed`` member function: ``incorrect`` will not return more than this number of examples, and ``guessed`` will not return more than this number of true positive examples. This does *not* affect any of the numerical metrics (precision, recall, or f-measure) - max_fn_examples: The maximum number actual examples of false negatives to record. This affects the ``missed`` member function and the ``correct`` member function: ``missed`` will not return more than this number of examples, and ``correct`` will not return more than this number of true negative examples. This does *not* affect any of the numerical metrics (precision, recall, or f-measure) - chunk_label: A regular expression indicating which chunks should be compared. Defaults to ``'.*'`` (i.e., all chunks). :type _tp: list(Token) :ivar _tp: List of true positives :type _fp: list(Token) :ivar _fp: List of false positives :type _fn: list(Token) :ivar _fn: List of false negatives :type _tp_num: int :ivar _tp_num: Number of true positives :type _fp_num: int :ivar _fp_num: Number of false positives :type _fn_num: int :ivar _fn_num: Number of false negatives. cKst|_t|_t|_t|_t|_|jdd|_|jdd|_|jdd|_ |jdd|_ d|_ d|_ d|_ d|_d|_d|_d |_dS) NZmax_tp_examplesdZmax_fp_examplesZmax_fn_examples chunk_labelz.*rgF)set_correct_guessed_tp_fp_fngetZ_max_tpZ_max_fpZ_max_fn _chunk_label_tp_num_fp_num_fn_num_count _tags_correct _tags_total_measuresNeedUpdate)selfkwargsr r r __init__rs zChunkScore.__init__cCs^|jrZ|j|j@|_|j|j|_|j|j|_t|j|_t|j|_t|j|_ d|_dS)NF) rrrrrrlenrrr)rr r r _updateMeasuress   zChunkScore._updateMeasurescCs|jt||j|jO_|jt||j|jO_|jd7_d|_yt|}t|}Wntk rxf}}YnX|jt |7_|j t ddt ||D7_ dS)aU Given a correctly chunked sentence, score another chunked version of the same sentence. :type correct: chunk structure :param correct: The known-correct ("gold standard") chunked sentence. :type guessed: chunk structure :param guessed: The chunked sentence to be scored. Tcss|]\}}||krdVqdS)r#Nr ).0tgr r r sz#ChunkScore.score..N) r _chunksetsrrrrr ValueErrorrr!rsumzip)rcorrectguessedZ correct_tagsZ guessed_tagsr r r scores  zChunkScore.scorecCs|jdkrdS|j|jS)z Return the overall tag-based accuracy for all text that have been scored by this ``ChunkScore``, using the IOB (conll2000) tag encoding. :rtype: float rr#)rr)rr r r rs zChunkScore.accuracycCs.|j|j|j}|dkr dS|j|SdS)z Return the overall precision for all texts that have been scored by this ``ChunkScore``. :rtype: float rN)r"rr)rdivr r r precisions  zChunkScore.precisioncCs.|j|j|j}|dkr dS|j|SdS)z Return the overall recall for all texts that have been scored by this ``ChunkScore``. :rtype: float rN)r"rr)rr/r r r recalls  zChunkScore.recall?cCsD|j|j}|j}|dks(|dkr,dSd||d||S)a Return the overall F measure for all texts that have been scored by this ``ChunkScore``. :param alpha: the relative weighting of precision and recall. Larger alpha biases the score towards the precision value, while smaller alpha biases the score towards the recall value. ``alpha`` should have a value in the range [0,1]. :type alpha: float :rtype: float rr#)r"r0r1)ralphaprr r r f_measures zChunkScore.f_measurecCs |jt|j}dd|DS)z Return the chunks which were included in the correct chunk structures, but not in the guessed chunk structures, listed in input order. :rtype: list of chunks cSsg|] }|dqS)r#r )r$cr r r sz%ChunkScore.missed..)r"listr)rchunksr r r misseds zChunkScore.missedcCs |jt|j}dd|DS)z Return the chunks which were included in the guessed chunk structures, but not in the correct chunk structures, listed in input order. :rtype: list of chunks cSsg|] }|dqS)r#r )r$r7r r r r8sz(ChunkScore.incorrect..)r"r9r)rr:r r r incorrects zChunkScore.incorrectcCst|j}dd|DS)z Return the chunks which were included in the correct chunk structures, listed in input order. :rtype: list of chunks cSsg|] }|dqS)r#r )r$r7r r r r8sz&ChunkScore.correct..)r9r)rr:r r r r,s zChunkScore.correctcCst|j}dd|DS)z Return the chunks which were included in the guessed chunk structures, listed in input order. :rtype: list of chunks cSsg|] }|dqS)r#r )r$r7r r r r8sz&ChunkScore.guessed..)r9r)rr:r r r r-s zChunkScore.guessedcCs|j|j|jS)N)r"rr)rr r r __len__szChunkScore.__len__cCsdtt|dS)z` Return a concise representation of this ``ChunkScoring``. :rtype: str z)reprr!)rr r r __repr__szChunkScore.__repr__cCs\dd|jdddd|jdddd|jdddd|jddd S) a- Return a verbose representation of this ``ChunkScoring``. This representation includes the precision, recall, and f-measure scores. For other information about the score, use the accessor methods (e.g., ``missed()`` and ``incorrect()``). :rtype: str zChunkParse score: z IOB Accuracy: r z5.1fz%% z Precision: z Recall: z F-Measure: z%%)rr0r1r6)rr r r __str__szChunkScore.__str__N)r2)__name__ __module__ __qualname____doc__r r"r.rr0r1r6r;r<r,r-r=r?r@r r r r r 3s=       r cCshd}g}xV|D]N}t|trTtj||jrB|j||f|jf|t|j7}q|d7}qWt |S)Nrr#) isinstancerrematchlabelappendfreezer!Zleavesr)r%countrposr:childr r r r(2s   r(NPS/c Cs.tjd}t|gg}x|j|D]}|j} | ddkr~t|dkrZtd|jdt|g} |d j| |j| q"| ddkrt|dkrtd |jd|j q"|d kr|d j| q"t | |\} } |r|rt ||| } |dj| | fq"Wt|dkr&td t|d|dS)aB Divide a string of bracketted tagged text into chunks and unchunked tokens, and produce a Tree. Chunks are marked by square brackets (``[...]``). Words are delimited by whitespace, and each word should have the form ``text/tag``. Words that do not contain a slash are assigned a ``tag`` of None. :param s: The string to be converted :type s: str :param chunk_label: The label to use for chunk nodes :type chunk_label: str :param root_label: The label to use for the root of the tree :type root_label: str :rtype: Tree z\[|\]|[^\[\]\s]+r[r#zUnexpected [ at char d]zUnexpected ] at char NzExpected ] at char rUrU) rFcompilerfinditergroupr!r)startrIpoprr) sr root_labelsepZ source_tagsetZ target_tagsetZWORD_OR_BRACKETstackrGtextchunkwordtagr r r tagstr2tree?s.          rcz(\S+)\s+(\S+)\s+([IOB])-?(\S+)?PPVPc Cst|gg}xt|jdD]\}}|js.qtj|}|dkrPtd|d|j\}}} } |dk rt| |krtd} | dko| |d jk} | dks| rt |d kr|j | d ks| rt| g} |d j | |j | |dj ||fqW|d S)a* Return a chunk structure for a single sentence encoded in the given CONLL 2000 style string. This function converts a CoNLL IOB string into a tree. It uses the specified chunk types (defaults to NP, PP and VP), and creates a tree rooted at a node labeled S (by default). :param s: The CoNLL string to be converted. :type s: str :param chunk_types: The chunk types to be converted. :type chunk_types: tuple :param root_label: The node label to use for the root. :type root_label: str :rtype: Tree  NzError on line rROIr#ZBOrTBrrUrUrU) r enumeratesplitstrip_LINE_RErGr)groupsrHr!rZrI) r[ chunk_typesr\r^linenolinerGrarbstateZ chunk_typeZ mismatch_Ir`r r r conllstr2treeus(       rsc Csg}x|D]}yP|j}d}x>|D]6}t|tr8td|j|d|d||fd}q"WWq tk r|j|d|ddfYq Xq W|S)z Return a list of 3-tuples containing ``(word, tag, IOB-tag)``. Convert a tree to the CoNLL IOB tag format. :param t: The tree to be converted. :type t: Tree :rtype: list(tuple) zB-z7Tree is too deeply nested to be printed in CoNLL formatrr#zI-rg)rHrErr)rIAttributeError)r%tagsrMcategoryprefixcontentsr r r rs     "rFcCst|g}x |D]\}}}|dkrD|r4tdn|j||fq|jdrn|jt|dd||fgq|jdrt|dkst|d t s|d j|ddkr|rtdq|jt|dd||fgn|d j||fq|dkr|j||fqtd |qW|S) z1 Convert the CoNLL IOB format to a tree. NzBad conll tag sequencezB-rTzI-rr#rgzBad conll tag rUrUrU)rr)rI startswithr!rErH)Zsentenceror\stricttreeraZpostagZchunktagr r r conlltags2trees(         r|cCsddt|D}dj|S)z Return a multiline string where each line contains a word, tag and IOB tag. Convert a tree to the CoNLL IOB string format :param t: The tree to be converted. :type t: Tree :rtype: str cSsg|]}dj|qS) )join)r$tokenr r r r8sz!tree2conllstr..rf)rr~)r%linesr r r tree2conllstrs ra\s*(\s*(?P.+?)\s*\s*)?(\s*(?P.+?)\s*\s*)?(\s*(?P.+?)\s*\s*)?\s*(\s*(?P.+?)\s*\s*)?(?P.*?)\s*\s*\s*z#]*?type="(?P\w+)"cCs t|gg}|dkrgSxtjd|D]}|j}yv|jdrtj|}|dkrZtd|t|jdg}|d j||j|n"|jdr|j n|d j|Wq&t t fk r}zt d|j dd |WYdd}~Xq&Xq&Wt |dkrt d |d S)Nz<[^>]+>|[^\s<]+z sB     2  5 # # 0