3
��]b�i � / @ s� d Z ddlZddlZyddlmZ W n ek
rD ddlmZ Y nX ddlmZ ddl m
Z
ddl m Z m Z ddl m
Z
mZ ye W n ek
r� eZY nX ye W n ek
r� eZY nX ye W n ek
r� eZY nX ye W n ek
�r
eefZY nX dd d
d d d
dgZejdejejB �jZejdej�jZ ejdej �j Z ejdej�j Z!ejdej�j Z"ejdej�j Z#dd� Z$ejd�jZ%ejdejejB �Z&ej'd�Z(ej'dde id �Z)G d d
� d
e*�Z+e+� Z,e,j-Z-ejd ej�ejd ej�gZ.d d!d"d#d$d%gZ/ejd&ej�ejd'ej�ejd(�gZ0d)gZ1e.e/e0e1fd*d �Z2d+d,� Z3d-d � Z4e2j e4_ d!d d"gZ5d.gZ6d/e5e6ed0�fd1d
�Z7d2d� Z8d3d4� Z9ejd5ej�Z:d6d7� Z;dS )8zcA cleanup tool for HTML.
Removes unwanted tags and content. See the `Cleaner` class for
details.
� N)�urlsplit)�etree)�defs)�
fromstring�XHTML_NAMESPACE)�
xhtml_to_html�_transform_result�
clean_html�clean�Cleaner�autolink�
autolink_html�
word_break�word_break_htmlzexpression\s*\(.*?\)z
@\s*importz </?[a-zA-Z]+|\son[a-zA-Z]+\s*=z^data:image/(.+);base64,z:(javascript|jscript|livescript|vbscript|data|about|mocha):z (xml|svg)c C s: d}x t | �D ]}d}t|�rdS qW |r.dS tt| ��S )NFT)�_find_image_dataurls�_is_unsafe_image_type�bool� _is_possibly_malicious_scheme)�sZ is_image_urlZ
image_type� r � /usr/lib64/python3.6/clean.py�_is_javascript_schemeT s r z [\s\x00-\x08\x0B\x0C\x0E-\x19]+z\[if[\s\n\r]+.*?][\s\n\r]*>z descendant-or-self::*[@style]z�descendant-or-self::a [normalize-space(@href) and substring(normalize-space(@href),1,1) != '#'] |descendant-or-self::x:a[normalize-space(@href) and substring(normalize-space(@href),1,1) != '#']�x)Z
namespacesc @ s� e Zd ZdZdZdZdZdZdZdZ dZ
dZ dZ dZ
dZdZdZdZdZdZdZdZejZdZf Zeddg�Zdd� Z e d d
d d gd d d d
d
�Z dd� Z dd� Z dd� Z!dd� Z"dd� Z#d!dd�Z$dd� Z%e&j'd e&j(�j)Z*d d � Z+d d � Z,dS )"r a
Instances cleans the document of each of the possible offending
elements. The cleaning is controlled by attributes; you can
override attributes in a subclass, or set them in the constructor.
``scripts``:
Removes any ``<script>`` tags.
``javascript``:
Removes any Javascript, like an ``onclick`` attribute. Also removes stylesheets
as they could contain Javascript.
``comments``:
Removes any comments.
``style``:
Removes any style tags.
``inline_style``
Removes any style attributes. Defaults to the value of the ``style`` option.
``links``:
Removes any ``<link>`` tags
``meta``:
Removes any ``<meta>`` tags
``page_structure``:
Structural parts of a page: ``<head>``, ``<html>``, ``<title>``.
``processing_instructions``:
Removes any processing instructions.
``embedded``:
Removes any embedded objects (flash, iframes)
``frames``:
Removes any frame-related tags
``forms``:
Removes any form tags
``annoying_tags``:
Tags that aren't *wrong*, but are annoying. ``<blink>`` and ``<marquee>``
``remove_tags``:
A list of tags to remove. Only the tags will be removed,
their content will get pulled up into the parent tag.
``kill_tags``:
A list of tags to kill. Killing also removes the tag's content,
i.e. the whole subtree, not just the tag itself.
``allow_tags``:
A list of tags to include (default include all).
``remove_unknown_tags``:
Remove any tags that aren't standard parts of HTML.
``safe_attrs_only``:
If true, only include 'safe' attributes (specifically the list
from the feedparser HTML sanitisation web site).
``safe_attrs``:
A set of attribute names to override the default list of attributes
considered 'safe' (when safe_attrs_only=True).
``add_nofollow``:
If true, then any <a> tags will have ``rel="nofollow"`` added to them.
``host_whitelist``:
A list or set of hosts that you can use for embedded content
(for content like ``<object>``, ``<link rel="stylesheet">``, etc).
You can also implement/override the method
``allow_embedded_url(el, url)`` or ``allow_element(el)`` to
implement more complex rules for what can be embedded.
Anything that passes this test will be shown, regardless of
the value of (for instance) ``embedded``.
Note that this parameter might not work as intended if you do not
make the links absolute before doing the cleaning.
Note that you may also need to set ``whitelist_tags``.
``whitelist_tags``:
A set of tags that can be included with ``host_whitelist``.
The default is ``iframe`` and ``embed``; you may wish to
include other tags like ``script``, or you may want to
implement ``allow_embedded_url`` for more control. Set to None to
include all tags.
This modifies the document *in place*.
TFN�iframe�embedc K sZ x:|j � D ].\}}t| |�s,td||f ��t| ||� q
W | jd krVd|krV| j| _d S )NzUnknown parameter: %s=%r� inline_style)�items�hasattr� TypeError�setattrr �style)�self�kw�name�valuer r r �__init__� s
zCleaner.__init__�src�href�code�object)�script�link�appletr r �layer�ac C s� t |d�r|j� }t|� x|jd�D ]
}d|_q&W | jsD| j|� t| jpNf �}t| j p\f �}t| j
pjf �}| j r~|j d� | j
r�t| j�}x:|jtj�D ]*}|j}x |j� D ]}||kr�||= q�W q�W | j�r*| j
o�| jtjk�s(x@|jtj�D ]0}|j}x$|j� D ]}|jd��r||= �qW q�W |j| jdd� | j�s�x\t|�D ]P}|jd�} td | �}
t d |
�}
| j |
��r�|jd= n|
| k�rJ|jd|
� �qJW | j �s*x�t |jd��D ]p}|jd
d �j � j!� d k�r�|j"� �q�|j#�p�d } td | �}
t d |
�}
| j |
��rd |_#n|
| k�r�|
|_#�q�W | j�s:| j$�rF|j tj%� | j$�rZ|j tj&� | j �rl|j d� | j�r�tj'|d� | j(�r�|j d
� nT| j �s�| j�r�xBt |jd
��D ]0}d|jdd �j � k�r�| j)|��s�|j"� �q�W | j*�r�|j d� | j+�r |j,d)� | j-�r�x\t |jd��D ]J}d} |j.� } x$| dk �rX| jd*k�rX| j.� } �q6W | dk�r$|j"� �q$W |j,d+� |j,d,� | j/�r�|j,tj0� | j1�r�|j d� |j,d-� | j2�r�|j,d.� g }
g }x`|j� D ]T}|j|k�r| j)|��r��q�|j3|� n&|j|k�r�| j)|��r"�q�|
j3|� �q�W |
�rb|
d" |k�rb|
j4d"�}d#|_|jj5� n8|�r�|d" |k�r�|j4d"�}|jdk�r�d#|_|j5� |j6� x|D ]}|j"� �q�W x|
D ]}|j7� �q�W | j8�r�|�r�t9d$��ttj:�}|�rlg }x(|j� D ] }|j|k�r|j3|� �qW |�rl|d" |k�rT|j4d"�}d#|_|jj5� x|D ]}|j7� �qZW | j;�r�xdt<|�D ]X}| j=|��s~|jd�}|�r�d%|k�r�d&d'| k�r��q~d(| }nd%}|jd|� �q~W dS )/z&
Cleans the document.
�getrootZimageZimgr* ZonF)Zresolve_base_hrefr � �typeztext/javascriptz
/* deleted */r+ Z
stylesheet�rel�meta�head�html�title�paramNr, r) r r r- Zform�button�input�select�textarea�blink�marqueer ZdivzIIt does not make sense to pass in both allow_tags and remove_unknown_tagsZnofollowz
nofollow z %s z %s nofollow)r4 r5 r6 )r, r) )r, )r r r- r) r7 )r8 r9 r: r; )r< r= )>r r/ r �iter�tag�comments�kill_conditional_comments�set� kill_tags� remove_tags�
allow_tags�scripts�add�safe_attrs_only�
safe_attrsr ZElement�attrib�keys�
javascriptr �
startswithZ
rewrite_links�_remove_javascript_linkr �_find_styled_elements�get�_replace_css_javascript�_replace_css_import�_has_sneaky_javascriptr �list�lower�strip� drop_tree�text�processing_instructions�CommentZProcessingInstructionZstrip_attributes�links�
allow_elementr3 �page_structure�update�embeddedZ getparent�framesZ
frame_tags�forms�
annoying_tags�append�pop�clear�reverseZdrop_tag�remove_unknown_tags�
ValueErrorZtags� add_nofollow�_find_external_links� allow_follow)r! �doc�elrC rD rE rI rJ Zaname�old�newZ found_parent�parent�_removeZ_kill�badr2 r r r �__call__ s
zCleaner.__call__c C s dS )zF
Override to suppress rel="nofollow" on some anchors.
Fr )r! �anchorr r r rk � s zCleaner.allow_followc C s� |j | jkrdS | j|j }t|ttf�r^x.|D ]&}|j|�}|sFdS | j||�s0dS q0W dS |j|�}|spdS | j||�S d S )NFT)r? �_tag_link_attrs�
isinstancerT �tuplerP �allow_embedded_url)r! rm �attrZone_attr�urlr r r r\ � s
zCleaner.allow_elementc C s^ | j d k r|j| j krdS t|�\}}}}}|j� jdd�d }|dkrLdS || jkrZdS dS ) NF�:� r �http�httpsT)r} r~ )�whitelist_tagsr? r rU �split�host_whitelist)r! rm rz �schemeZnetloc�pathZqueryZfragmentr r r rx � s
zCleaner.allow_embedded_urlc C s g }| j |dd� tj� dS )z�
IE conditional comments basically embed HTML that the parser
doesn't normally see. We can't allow anything like that, so
we'll kill any comments that could be conditional.
c S s t j| j�S )N)�_conditional_comment_re�searchrX )rm r r r �<lambda>� s z3Cleaner.kill_conditional_comments.<locals>.<lambda>N)�_kill_elementsr rZ )r! rl rr r r r rA � s z!Cleaner.kill_conditional_commentsc C sD g }x$|j |�D ]}||�r|j|� qW x|D ] }|j� q0W d S )N)r> rc rW )r! rl Z conditionZiteraterr rm r r r r� � s
zCleaner._kill_elementsc C s t d|�}t|�rdS |S )Nr0 )�_substitute_whitespacer )r! r+ ro r r r rN � s
z Cleaner._remove_javascript_linkz /\*.*?\*/c C sj | j d|�}|jdd�}td|�}|j� }d|kr6dS d|krBdS d|krNdS d|krZdS t|�rfdS dS ) a�
Depending on the browser, stuff like ``e x p r e s s i o n(...)``
can get interpreted, or ``expre/* stuff */ssion(...)``. This
checks for attempt to do stuff like this.
Typically the response will be to kill the entire style; if you
have just a bit of Javascript in the style another rule will catch
that and remove only the Javascript from the style; this catches
more sneaky attempts.
r0 �\z javascript:Tz expression(z@importz
</noscriptF)�_substitute_comments�replacer� rU �_looks_like_tag_content)r! r r r r rS � s
z Cleaner._has_sneaky_javascriptc C s8 t |�}t|t�r t|�}n
tj|�}| |� t||�S )N)r1 rv �
basestringr �copy�deepcopyr )r! r5 � result_typerl r r r r s
zCleaner.clean_html)N)-�__name__�
__module__� __qualname__�__doc__rF rL r@ r r r[ r3 r] rY r_ r` ra rb rD rE rC rg rH r rI ri r� rB r r% �dictru rs rk r\ rx rA r� rN �re�compile�S�subr� rS r r r r r r m sV ] 5
zb(?P<body>https?://(?P<host>[a-z0-9._-]+)(?:/[/\-_.,a-z0-9%&?;=~]*)?(?:\([/\-_.,a-z0-9%&?;=~]*\))?)z9mailto:(?P<body>[a-z0-9._-]+@(?P<host>[a-z0-9_.-]+[a-z]))r; Zprer( r4 r: r. z
^localhostz\bexample\.(?:com|org|net)$z^127\.0\.0\.1$Znolinkc C s� | j |krdS | jd�}|r>|j� }x|D ]}||kr*dS q*W xht| �D ]\}t|||||d� |jrHt|j||| jd�\}} | rH||_| j|�}
| | |
d |
d �< qHW | j r�t| j ||| jd�\}} | r�|| _ | | dd�< dS )a
Turn any URLs into links.
It will search for links identified by the given regular
expressions (by default mailto and http(s) links).
It won't link text in an element in avoid_elements, or an element
with a class in avoid_classes. It won't link to anything with a
host that matches one of the regular expressions in avoid_hosts
(default localhost and 127.0.0.1).
If you pass in an element, the element's tail will not be
substituted, only the contents of the element.
N�class)� link_regexes�avoid_elements� avoid_hosts�
avoid_classes)�factoryr| r )
r? rP r� rT r �tail�
_link_textZ makeelement�indexrX ) rm r� r� r� r� �
class_nameZ match_class�childrX Z
tail_childrenr� Z pre_childrenr r r r 4 s4
c C s� d}g }d}�x�d \}}x�|D ]~} |}
xJ| j | |
d�} | d kr@P | jd�} x"|D ]}
|
j | �rP| j� }
P qPW P q(W | d kr|q |d ks�| j� |k r | }| j� }q W |d kr�|r�|d j s�t�| |d
_n| s�t�| }P |jd�}|j� }|jd��s|jd��r|d8 }|d d� }| d |j� � }|�rJ|d j �s>t�||d _n| �sVt�|}|d�}|jd |� |jd
�}|�s�|}|jd��s�|jd��r�|d d� }||_|j |� | |d � } qW ||fS )Nr0 r )�pos�hostr| �.�,r. r' �body)NN���r� r� r� r� r� )
r� �group�end�startr� �AssertionError�endswithrB rX rc )rX r� r� r� Z leading_textr[ Zlast_posZ
best_matchZbest_posZregexZ regex_pos�matchr� Z
host_regexr+ r� Z prev_textrt r� r r r r� a sf
r� c O s@ t | �}t| t�r t| �}n
tj| �}t|f|�|� t||�S )N)r1 rv r� r r� r� r r )r5 �argsr" r� rl r r r r
� s
Znobreak�( i c C s� | j tkrdS | jd�}|rLd}|j� }x|D ]}||kr.d}P q.W |rLdS | jrbt| j||�| _x4| D ],}t|||||d� |jrht|j||�|_qhW dS )a�
Breaks any long words found in the body of the text (not attributes).
Doesn't effect any of the tags in avoid_elements, by default
``<textarea>`` and ``<pre>``
Breaks words by inserting ​, which is a unicode character
for Zero Width Space character. This generally takes up no space
in rendering, but does copy as a space, and in monospace contexts
usually takes up space.
See http://www.cs.tut.fi/~jkorpela/html/nobr.html for a discussion
Nr� FT)� max_widthr� r� �break_character)r? �_avoid_word_break_elementsrP r� rX � _break_textr r� ) rm r� r� r� r� r� Z
dont_breakZavoidr� r r r r � s*
c O s* t | �}t| �}t|f|�|� t||�S )N)r1 r r r )r5 r� r" r� rl r r r r � s c C s> | j � }x0|D ](}t|�|krt|||�}| j||�} qW | S )N)r� �len�
_insert_breakr� )rX r� r� Zwords�wordZ replacementr r r r� � s
r� z[^a-z]c C s� | }d}xrt | �|krz| d |� }ttj|��}|r\|d }|j� |d kr\| d |j� � }||| 7 }| t |�d � } q
W || 7 }|S )Nr0 r| �
r� )r� rT �_break_prefer_re�finditerr� )r� �widthr� Z orig_word�resultr� ZbreaksZ
last_breakr r r r� � s r� )<r� r� r� Zurlparser � ImportErrorZ urllib.parseZlxmlr Z lxml.htmlr r r r r Zunichr� NameError�chrZunicode�str�bytesr� �__all__r� r� �Ir� rQ rR �ASCIIr� r� �findallr r r r r� r� ZXPathrO rj r) r r
r Z
_link_regexesZ_avoid_elementsZ _avoid_hostsZ_avoid_classesr r� r
r� Z_avoid_word_break_classesr r r� r� r� r r r r �<module> s�
3 *:
&