3 d;@sddlZddlZddlZddlZddlmZddlmZmZm Z ddl m Z m Z ddl mZmZddZdd Zd d Zd d ZddZddZddZddZddZddZddZddZd>d$d%Zd&d'Zd?d(d)Zed@dAgZedBdCdDdEdFdGdHdIdJg Z d;d<Z!e"d=kredS)KN)treebank)BrillTaggerTrainer RegexpTagger UnigramTagger)PosWord)Template error_listcCs tdS)z Run a demo with defaults. See source comments for details, or docstrings of any of the more specific demo_* functions. N)postagr r -/tmp/pip-build-v9q4h5k9/nltk/nltk/tbl/demo.pydemosr cCstdddS)zN Exemplify repr(Rule) (see also str(Rule) and Rule.format("verbose")) repr) ruleformatN)r r r r r demo_repr_rule_formatsrcCstdddS)zN Exemplify repr(Rule) (see also str(Rule) and Rule.format("verbose")) str)rN)r r r r r demo_str_rule_format$srcCstdddS)z* Exemplify Rule.format("verbose") verbose)rN)r r r r r demo_verbose_rule_format+srcCstttdddggddS) a The feature/s of a template takes a list of positions relative to the current word where the feature should be looked for, conceptually joined by logical OR. For instance, Pos([-1, 1]), given a value V, will hold whenever V is found one step to the left and/or one step to the right. For contiguous ranges, a 2-arg form giving inclusive end points can also be used: Pos(-3, -1) is the same as the arg below. ) templatesN)r rrr r r r demo_multiposition_feature2s rcCs$tttdgtddggddS)z8 Templates can have more than a single feature. rrr)rNrr)r rrrr r r r demo_multifeature_templateAsrcCstddddS)ah Show aggregate statistics per template. Little used templates are candidates for deletion, much used templates may possibly be refined. Deleting unused templates is mostly about saving time and/or space: training is basically O(T) in the number of templates T (also in terms of memory usage, which often will be the limiting factor). T)incremental_statstemplate_statsN)r r r r r demo_template_statisticsHs r cCsptjd ddgddgdd}tjd dddgddgdd}ttj||gdd}td jt|t|ddd d S)a  Template.expand and Feature.expand are class methods facilitating generating large amounts of templates. See their documentation for details. Note: training with 500 templates can easily fill all available even on relatively small corpora rrrF)Z excludezeroTr) combinationsz8Generated {} templates for transformation-based learning)rrrNrrr)rr) rexpandrlistrprintformatlenr )ZwordtplsZtagtplsrr r r demo_generated_templatesTs  r'cCstdddddS)z Plot a learning curve -- the contribution on tagging accuracy of the individual rules. Note: requires matplotlib Tzlearningcurve.png)rseparate_baseline_datalearning_curve_outputN)r r r r r demo_learning_curvehsr*cCstdddS)zW Writes a file with context for each erroneous word after tagging testing data z errors.txt) error_outputN)r r r r r demo_error_analysisusr,cCstdddS)zm Serializes the learned tagger to a file in pickle format; reloads it and validates the process. z tagger.pcl)serialize_outputN)r r r r r demo_serialize_tagger|sr.cCstdddddS)z Discard rules with low accuracy. This may hurt performance a bit, but will often produce rules which are more interesting read to a human. i gQ? ) num_sentsmin_acc min_scoreN)r r r r r demo_high_accuracy_rulessr3,r皙?Frc&&Cs|pt}|dkr&ddlm}m}|}t|||||\}}}}|rtjj|st||d}t |d}t j ||WdQRXt dj |t |}t j|}t d|WdQRXnt||d}t d|rt d j |j|tj}t|||| d }t d |j||||}t d tj|d d|rFt d|j||dkrt dx8t|jdD]&\}}t |dd|j | dqhW| rt d|j||\} }!t d|st d|j}"| r|j|!|r$t||!|"|dt d|n t d|j|} | r$|j| dk rt | d4}#|#jd| |#jdjt|| jddWdQRXt d| | dk r|j|} t | d}t j ||WdQRXt d| t | }t j|}$WdQRXt d| |j|}%| |%krt d nt d!dS)"a Brill Tagger Demonstration :param templates: how many sentences of training and testing data to use :type templates: list of Template :param tagged_data: maximum number of rule instances to create :type tagged_data: C{int} :param num_sents: how many sentences of training and testing data to use :type num_sents: C{int} :param max_rules: maximum number of rule instances to create :type max_rules: C{int} :param min_score: the minimum score for a rule in order for it to be considered :type min_score: C{int} :param min_acc: the minimum score for a rule in order for it to be considered :type min_acc: C{float} :param train: the fraction of the the corpus to be used for training (1=all) :type train: C{float} :param trace: the level of diagnostic tracing output to produce (0-4) :type trace: C{int} :param randomize: whether the training data should be a random subset of the corpus :type randomize: C{bool} :param ruleformat: rule output format, one of "str", "repr", "verbose" :type ruleformat: C{str} :param incremental_stats: if true, will tag incrementally and collect stats for each rule (rather slow) :type incremental_stats: C{bool} :param template_stats: if true, will print per-template statistics collected in training and (optionally) testing :type template_stats: C{bool} :param error_output: the file where errors will be saved :type error_output: C{string} :param serialize_output: the file where the learned tbl tagger will be saved :type serialize_output: C{string} :param learning_curve_output: filename of plot of learning curve(s) (train and also test, if available) :type learning_curve_output: C{string} :param learning_curve_take: how many rules plotted :type learning_curve_take: C{int} :param baseline_backoff_tagger: the file where rules will be saved :type baseline_backoff_tagger: tagger :param separate_baseline_data: use a fraction of the training data exclusively for training baseline :type separate_baseline_data: C{bool} :param cache_baseline_tagger: cache baseline tagger to this file (only interesting as a temporary workaround to get deterministic output from the baseline unigram tagger between python versions) :type cache_baseline_tagger: C{string} Note on separate_baseline_data: if True, reuse training data both for baseline and rule learner. This is fast and fine for a demo, but is likely to generalize worse on unseen data. Also cannot be sensibly used for learning curves on training data (the baseline will be artificially high). Nr)brill24describe_template_sets)backoffwz)Trained baseline tagger, pickled it to {}zReloaded pickled tagger from zTrained baseline taggerz! Accuracy on test set: {:0.4f})rzTraining tbl tagger...zTrained tbl tagger in z0.2fz secondsz Accuracy on test set: %.4frz Learned rules: Z4d szJIncrementally tagging the test data, collecting individual rule statisticsz Rule statistics collectedzbWARNING: train_stats asked for separate_baseline_data=True; the baseline will be artificially high)takez Wrote plot of learning curve to zTagging the test datazErrors for Brill Tagger %r  zutf-8z)Wrote tagger errors including context to zWrote pickled tagger to z4Reloaded tagger tried on test set, results identicalz;PROBLEM: Reloaded tagger gave different results on test set) REGEXP_TAGGERnltk.tag.brillr7r8_demo_prepare_dataospathexistsropenpickledumpr$r%loadZaccuracytimertrain enumeraterulesZbatch_tag_incrementalZ train_statsZprint_template_statistics _demo_plotZ tag_sentswritejoinr encode)&r tagged_datar0Z max_rulesr2r1rJtrace randomizerrrr+r-r)Zlearning_curve_takeZbaseline_backoff_taggerr(Zcache_baseline_taggerr7r8 training_data baseline_data gold_data testing_dataZbaseline_taggerZ print_rulesZtbrillZtrainerZ brill_taggerZrulenoZruleZ taggedtest teststats trainstatsfZbrill_tagger_reloadedZtaggedtest_reloadedr r r r sX       "     *       r cCs2|dkrtdtj}|dks,t||kr4t|}|rPtjt|tj|t||}|d|}|||}dd|D}|s|} n&t|d} |d| || d} }t|\} } t|\} }t| \}}td| dd|ddtd | dd| ddtd j |||rd nd || ||fS) Nz%Loading tagged data from treebank... cSsg|]}dd|DqS)cSsg|] }|dqS)rr ).0tr r r asz1_demo_prepare_data...r )r[sentr r r r]asz&_demo_prepare_data..rzRead testing data (dz sents/z wds)zRead training data (z-Read baseline data ({:d} sents/{:d} wds) {:s}z[reused the training set]) r$rZ tagged_sentsr&randomseedshuffleint corpus_sizer%)rQrJr0rSr(cutoffrTrVrWrUZ bl_cutoffZ trainseqsZ traintokensZtestseqsZ testtokensZ bltrainseqsZ bltraintokensr r r rAQs8         rAc sdg}x"dD]}|j|d |qWfdd|d|D}dg}x"dD]}|j|d |q\Wfdd|d|D}ddlj}ttt|}|j|||||jddddg|j|dS) NZ initialerrorsZ rulescoresrcsg|]}d|dqS)r tokencountr )r[x)rXr r r]}sz_demo_plot..csg|]}d|dqS)rrgr )r[rh)rYr r r]srg?rr) appendZmatplotlib.pyplotZpyplotr#ranger&ZplotZaxisZsavefig) r)rXrYr=Z testcurveZ rulescoreZ traincurveZpltrr )rXrYr rMys   rM^-?[0-9]+(\.[0-9]+)?$CD.*NN(The|the|A|a|An|an)$AT.*able$JJ.*ness$.*ly$RB.*s$NNS.*ing$VBG.*ed$VBDcCst|tdd|DfS)Ncss|]}t|VqdS)N)r&)r[rhr r r szcorpus_size..)r&sum)Zseqsr r r resre__main__)NNr4r5rNr6rFrFFNNNr5NFN)NN)rlrm)rnro)rlrm)rprq)rrrs)rtro)rurv)rwrx)ryrz)r{r|)rnro)#rBrFrarIZ nltk.corpusrZnltk.tagrrrr@rrZnltk.tblrr r rrrrrr r'r*r,r.r3r rArMZ NN_CD_TAGGERr?re__name__r r r r  sn     3(