JezK
Edit File: charsetprober.cpython-311.pyc
� � �f� � �> � d dl Z d dlZddlmZ G d� de� � ZdS )� N� )� ProbingStatec � � e Zd ZdZd d�Zd� Zed� � � Zd� Zed� � � Z d� Z e d � � � Z e d � � � Z e d � � � ZdS ) � CharSetProbergffffff�?Nc �^ � d | _ || _ t j t � � | _ d S �N)�_state� lang_filter�logging� getLogger�__name__�logger)�selfr s �H/opt/alt/python311/lib/python3.11/site-packages/chardet/charsetprober.py�__init__zCharSetProber.__init__' s' � ��� �&��� �'��1�1�� � � � c �( � t j | _ d S r )r � DETECTINGr �r s r �resetzCharSetProber.reset, s � �"�,�� � � r c � � d S r � r s r � charset_namezCharSetProber.charset_name/ s � ��tr c � � d S r r )r �bufs r �feedzCharSetProber.feed3 s � � �r c � � | j S r )r r s r �statezCharSetProber.state6 s � ��{�r c � � dS )Ng r r s r �get_confidencez CharSetProber.get_confidence: s � ��sr c �2 � t j dd| � � } | S )Ns ([ -])+� )�re�sub)r s r �filter_high_byte_onlyz#CharSetProber.filter_high_byte_only= s � ��f�&��c�2�2��� r c � � t � � }t j d| � � }|D ]Z}|� |dd� � � |dd� }|� � � s|dk rd}|� |� � �[|S )u9 We define three types of bytes: alphabet: english alphabets [a-zA-Z] international: international characters [-ÿ] marker: everything else [^a-zA-Z-ÿ] The input buffer can be thought to contain a series of words delimited by markers. This function works to filter all words that contain at least one international character. All contiguous sequences of markers are replaced by a single space ascii character. This filter applies to all scripts which do not use English characters. s% [a-zA-Z]*[�-�]+[a-zA-Z]*[^a-zA-Z�-�]?N���� �r" )� bytearrayr# �findall�extend�isalpha)r �filtered�words�word� last_chars r �filter_international_wordsz(CharSetProber.filter_international_wordsB s� � � �;�;�� � �O� � � �� � '� '�D� �O�O�D��"��I� &� &� &� �R�S�S� �I� �$�$�&�&� !�9�w�+>�+>� � � �O�O�I� &� &� &� &��r c � � t � � }d}d}t t | � � � � D ]y}| ||dz � }|dk rd}n|dk rd}|dk rS|� � � s?||k r4|s2|� | ||� � � |� d� � |dz }�z|s |� | |d � � � |S ) a� Returns a copy of ``buf`` that retains only the sequences of English alphabet and high byte characters that are not between <> characters. Also retains English alphabet and high byte characters immediately before occurrences of >. This filter can be applied to all scripts which contain both English characters and extended ASCII characters, but is currently only used by ``Latin1Prober``. Fr r � >� <Tr( r" N)r) �range�lenr, r+ )r r- �in_tag�prev�curr�buf_chars r �filter_with_english_lettersz)CharSetProber.filter_with_english_lettersg s� � � �;�;�������#�c�(�(�O�O� � �D��4��q��=�)�H��4� � � ����T�!�!� �� �'�!�!�(�*:�*:�*<�*<�!��$�;�;�v�;� �O�O�C��T� �N�3�3�3� �O�O�D�)�)�)��a�x��� � (� �O�O�C����J� '� '� '��r r )r � __module__� __qualname__�SHORTCUT_THRESHOLDr r �propertyr r r r � staticmethodr% r1 r; r r r r r # s� � � � � � � ��2� 2� 2� 2� -� -� -� �� � �X�� � � � �� � �X��� � � �� � �\�� �"� "� �\�"�H �)� )� �\�)� )� )r r )r r# �enumsr �objectr r r r �<module>rC si ��: ���� � � � � � � � � � �n� n� n� n� n�F� n� n� n� n� nr