3 d!@sdZddlmZddlmZmZmZddlmZm Z ddl m Z m Z ddl mZddd Zd d Zd d ZGdddZdddZdS)z Utility functions for parsers. )load)CFGPCFGFeatureGrammar)Chart ChartParser) FeatureChartFeatureChartParser)InsideChartParserNcKst|f|}t|tstdt|trB|dkr4t}||||dSt|trr|dkrXt}|dkrdt}||||dS|dkr~t }|dkrt }||||dSdS)a Load a grammar from a file, and build a parser based on that grammar. The parser depends on the grammar format, and might also depend on properties of the grammar itself. The following grammar formats are currently supported: - ``'cfg'`` (CFGs: ``CFG``) - ``'pcfg'`` (probabilistic CFGs: ``PCFG``) - ``'fcfg'`` (feature-based CFGs: ``FeatureGrammar``) :type grammar_url: str :param grammar_url: A URL specifying where the grammar is located. The default protocol is ``"nltk:"``, which searches for the file in the the NLTK data package. :type trace: int :param trace: The level of tracing that should be used when parsing a text. ``0`` will generate no tracing output; and higher numbers will produce more verbose tracing output. :param parser: The class used for parsing; should be ``ChartParser`` or a subclass. If None, the class depends on the grammar format. :param chart_class: The class used for storing the chart; should be ``Chart`` or a subclass. Only used for CFGs and feature CFGs. If None, the chart class depends on the grammar format. :type beam_size: int :param beam_size: The maximum length for the parser's edge queue. Only used for probabilistic CFGs. :param load_args: Keyword parameters used when loading the grammar. See ``data.load`` for more information. z1The grammar must be a CFG, or a subclass thereof.N)trace beam_size)r chart_class) r isinstancer ValueErrorrr rr rrr)Z grammar_urlr parserr r Z load_argsgrammarr//tmp/pip-build-v9q4h5k9/nltk/nltk/parse/util.py load_parsers$"    rc csRxLt|ddD]<\}\}}t||d||dddddg }dj|d}|VqWdS) a A module to convert a single POS tagged sentence into CONLL format. >>> from nltk import word_tokenize, pos_tag >>> text = "This is a foobar sentence." >>> for line in taggedsent_to_conll(pos_tag(word_tokenize(text))): ... print(line, end="") 1 This _ DT DT _ 0 a _ _ 2 is _ VBZ VBZ _ 0 a _ _ 3 a _ DT DT _ 0 a _ _ 4 foobar _ JJ JJ _ 0 a _ _ 5 sentence _ NN NN _ 0 a _ _ 6 . _ . . _ 0 a _ _ :param sentence: A single input sentence to parse :type sentence: list(tuple(str, str)) :rtype: iter(str) :return: a generator yielding a single sentence in CONLL format. )start_0a  N) enumeratestrjoin)sentenceiwordtagZ input_strrrrtaggedsent_to_conllOsr#ccs&x |D]}t|EdHdVqWdS)a5 A module to convert the a POS tagged document stream (i.e. list of list of tuples, a list of sentences) and yield lines in CONLL format. This module yields one line per word and two newlines for end of sentence. >>> from nltk import word_tokenize, sent_tokenize, pos_tag >>> text = "This is a foobar sentence. Is that right?" >>> sentences = [pos_tag(word_tokenize(sent)) for sent in sent_tokenize(text)] >>> for line in taggedsents_to_conll(sentences): ... if line: ... print(line, end="") 1 This _ DT DT _ 0 a _ _ 2 is _ VBZ VBZ _ 0 a _ _ 3 a _ DT DT _ 0 a _ _ 4 foobar _ JJ JJ _ 0 a _ _ 5 sentence _ NN NN _ 0 a _ _ 6 . _ . . _ 0 a _ _ 1 Is _ VBZ VBZ _ 0 a _ _ 2 that _ IN IN _ 0 a _ _ 3 right _ NN NN _ 0 a _ _ 4 ? _ . . _ 0 a _ _ :param sentences: Input sentences to parse :type sentence: list(list(tuple(str, str))) :rtype: iter(str) :return: a generator yielding sentences in CONLL format. Nz )r#) sentencesrrrrtaggedsents_to_conllis! r%c@s$eZdZdZdddZd ddZdS) TestGrammarz Unit tests for CFG. NcCs*||_t|dd|_||_||_||_dS)Nr)r )Z test_grammarrcpsuite_acceptZ_reject)selfrr(acceptrejectrrr__init__s zTestGrammar.__init__Fc Csx|jD]}t|ddddxd D]}x||D]}|j}t|jj|}|r||r|tt|x|D] }t|qlW|dkr|gkrtd|qd}q4|rtd |q4d} q4Wq&W|r| rtd qWd S) a} Sentences in the test suite are divided into two classes: - grammatical (``accept``) and - ungrammatical (``reject``). If a sentence should parse according to the grammar, the value of ``trees`` will be a non-empty list. If a sentence should be rejected according to the grammar, then the value of ``trees`` will be None. doc: )endr+r,zSentence '%s' failed to parse'TzSentence '%s' received a parse'zAll tests passed!N)r+r,)r(printsplitlistr'parser) r*Z show_treestestkeysenttokensZtreestreeacceptedZrejectedrrrruns(     zTestGrammar.run)NN)F)__name__ __module__ __qualname____doc__r-r<rrrrr&s r&#%;cCs|dk r|j|}g}x|jdD]}|dks"|d|kr s   8+1