3
0dG: @ s d Z dZdgZddlmZ ddlZddlZddlmZm Z m
Z
m Z m Z ddl
mZmZ ddlmZmZmZmZmZ d ZG d
d d eeZG d d deZdS )
zCUse the HTMLParser library to parse HTML files that aren't too bad.MITHTMLParserTreeBuilder )
HTMLParserN)CDataComment DeclarationDoctypeProcessingInstruction)EntitySubstitution
UnicodeDammit)DetectsXMLParsedAsHTMLParserRejectedMarkupHTMLHTMLTreeBuilderSTRICTz html.parserc @ s| e Zd ZdZdZdZdd Zdd Zdd Zd d d Z d d
dZ
dd Z dd Z dd Z
dd Zdd Zdd Zdd Zd S ) BeautifulSoupHTMLParserzA subclass of the Python standard library's HTMLParser class, which
listens for HTMLParser events and translates them into calls
to Beautiful Soup's tree construction API.
ignorereplacec O s4 |j d| j| _tj| f|| g | _| j dS )a Constructor.
:param on_duplicate_attribute: A strategy for what to do if a
tag includes the same attribute more than once. Accepted
values are: REPLACE (replace earlier values with later
ones, the default), IGNORE (keep the earliest value
encountered), or a callable. A callable must take three
arguments: the dictionary of attributes already processed,
the name of the duplicate attribute, and the most recent value
encountered.
on_duplicate_attributeN)popREPLACEr r __init__ already_closed_empty_elementZ_initialize_xml_detector)selfargskwargs r I/var/www/html/virt/lib/python3.6/site-packages/bs4/builder/_htmlparser.pyr . s
z BeautifulSoupHTMLParser.__init__c C s t |d S )N)r
)r messager r r errorJ s z BeautifulSoupHTMLParser.errorc C s | j ||dd}| j| dS )zHandle an incoming empty-element tag.
This is only called when the markup looks like .
:param name: Name of the tag.
:param attrs: Dictionary of the tag's attributes.
F)handle_empty_elementN)handle_starttag
handle_endtag)r nameattrstagr r r handle_startendtagZ s z*BeautifulSoupHTMLParser.handle_startendtagTc C s i }xh|D ]`\}}|dkr d}||kr^| j }|| jkr8qf|d| jfkrP|||< qf|||| n|||< d}q
W | j \} }
| jj|dd|| |
d} | r| jr|r| j|dd | jj | | j
dkr| j | dS )a3 Handle an opening tag, e.g. ''
:param name: Name of the tag.
:param attrs: Dictionary of the tag's attributes.
:param handle_empty_element: True if this tag is known to be
an empty-element tag (i.e. there is not expected to be any
closing tag).
N z"")
sourceline sourceposF)check_already_closed) r IGNOREr getpossoupr! Zis_empty_elementr" r appendZ _root_tagZ_root_tag_encountered) r r# r$ r Z attr_dictkeyvalueZon_dupe attrvaluer( r) r% r r r r! i s,
z'BeautifulSoupHTMLParser.handle_starttagc C s, |r || j kr | j j| n | jj| dS )zHandle a closing tag, e.g. ''
:param name: A tag name.
:param check_already_closed: True if this tag is expected to
be the closing portion of an empty-element tag,
e.g. ''.
N)r remover- r" )r r# r* r r r r" s z%BeautifulSoupHTMLParser.handle_endtagc C s | j j| dS )z4Handle some textual data that shows up between tags.N)r- handle_data)r datar r r r3 s z#BeautifulSoupHTMLParser.handle_datac C s |j dr t|jdd}n$|j dr8t|jdd}nt|}d}|dk rxR| jjdfD ]B}|sdqZyt|gj|}W qZ tk
r } zW Y dd}~X qZX qZW |sy t|}W n& t t
fk
r } zW Y dd}~X nX |pd}| j | dS )zHandle a numeric character reference by converting it to the
corresponding Unicode character and treating it as textual
data.
:param name: Character number, possibly in hexadecimal.
x XN z windows-1252u �)
startswithintlstripr- original_encoding bytearraydecodeUnicodeDecodeErrorchr
ValueError
OverflowErrorr3 )r r# Z real_namer4 encodinger r r handle_charref s*
z&BeautifulSoupHTMLParser.handle_charrefc C s0 t jj|}|dk r|}nd| }| j| dS )zHandle a named entity reference by converting it to the
corresponding Unicode character(s) and treating it as textual
data.
:param name: Name of the entity reference.
Nz&%s)r
ZHTML_ENTITY_TO_CHARACTERgetr3 )r r# characterr4 r r r handle_entityref s
z(BeautifulSoupHTMLParser.handle_entityrefc C s& | j j | j j| | j jt dS )zOHandle an HTML comment.
:param data: The text of the comment.
N)r- endDatar3 r )r r4 r r r handle_comment s
z&BeautifulSoupHTMLParser.handle_commentc C s6 | j j |tdd }| j j| | j jt dS )zYHandle a DOCTYPE declaration.
:param data: The text of the declaration.
zDOCTYPE N)r- rI lenr3 r )r r4 r r r handle_decl s
z#BeautifulSoupHTMLParser.handle_declc C sN |j jdr$t}|tdd }nt}| jj | jj| | jj| dS )z{Handle a declaration of unknown type -- probably a CDATA block.
:param data: The text of the declaration.
zCDATA[N)upperr9 r rK r r- rI r3 )r r4 clsr r r unknown_decl s
z$BeautifulSoupHTMLParser.unknown_declc C s0 | j j | j j| | j| | j jt dS )z\Handle a processing instruction.
:param data: The text of the instruction.
N)r- rI r3 Z_document_might_be_xmlr )r r4 r r r handle_pi s
z!BeautifulSoupHTMLParser.handle_piN)T)T)__name__
__module__ __qualname____doc__r+ r r r r& r! r" r3 rE rH rJ rL rO rP r r r r r $ s
7
(
r c sN e Zd ZdZdZdZeZeee gZ
dZ d fdd Z d ddZ
d d
Z ZS )
r zpA Beautiful soup `TreeBuilder` that uses the `HTMLParser` parser,
found in the Python standard library.
FTNc sp t }x&dD ] }||kr |j|}|||< q W tt| jf | |pFg }|pNi }|j| d|d< ||f| _dS )a Constructor.
:param parser_args: Positional arguments to pass into
the BeautifulSoupHTMLParser constructor, once it's
invoked.
:param parser_kwargs: Keyword arguments to pass into
the BeautifulSoupHTMLParser constructor, once it's
invoked.
:param kwargs: Keyword arguments for the superclass constructor.
r Fconvert_charrefsN)r )dictr superr r update parser_args)r rY Z
parser_kwargsr Zextra_parser_kwargsargr0 ) __class__r r r * s
z HTMLParserTreeBuilder.__init__c c s\ t |tr |dddfV dS |g}|g}||g}t|||d|d}|j|j|j|jfV dS )a Run any preliminary steps necessary to make incoming markup
acceptable to the parser.
:param markup: Some markup -- probably a bytestring.
:param user_specified_encoding: The user asked to try this encoding.
:param document_declared_encoding: The markup itself claims to be
in this encoding.
:param exclude_encodings: The user asked _not_ to try any of
these encodings.
:yield: A series of 4-tuples:
(markup, encoding, declared encoding,
has undergone character replacement)
Each 4-tuple represents a strategy for converting the
document to Unicode and parsing it. Each strategy will be tried
in turn.
NFT)known_definite_encodingsuser_encodingsZis_htmlexclude_encodings)
isinstancestrr markupr< Zdeclared_html_encodingZ contains_replacement_characters) r ra Zuser_specified_encodingZdocument_declared_encodingr^ r\ r] Z
try_encodingsZdammitr r r prepare_markupC s
z$HTMLParserTreeBuilder.prepare_markupc C sh | j \}}t||}| j|_y|j| W n* tk
rT } zt|W Y dd}~X nX |j g |_dS )z{Run some incoming markup through some parsing process,
populating the `BeautifulSoup` object in self.soup.
N)rY r r- feedAssertionErrorr
closer )r ra r r parserrD r r r rc t s
zHTMLParserTreeBuilder.feed)NN)NNN)rQ rR rS rT Zis_xmlZ picklable
HTMLPARSERNAMEr r featuresZTRACKS_LINE_NUMBERSr rb rc
__classcell__r r )r[ r r s
0)rT __license____all__ html.parserr syswarningsZ bs4.elementr r r r r Z
bs4.dammitr
r Z bs4.builderr r
r r r rg r r r r r r s z