D7net
Home
Console
Upload
information
Create File
Create Folder
About
Tools
:
/
lib
/
python3
/
dist-packages
/
s3transfer
/
__pycache__
/
Filename :
bandwidth.cpython-312.pyc
back
Copy
� �<�e= � � � d dl Z d dlZ G d� de� Z G d� d� Z G d� d� Z G d� d � Z G d � d � Z G d � d � Z G d� d� Z G d� d� Z y)� Nc � � � e Zd Z� fd�Z� xZS )�RequestExceededExceptionc �d �� || _ || _ dj ||� }t �| � |� y)a� Error when requested amount exceeds what is allowed The request that raised this error should be retried after waiting the time specified by ``retry_time``. :type requested_amt: int :param requested_amt: The originally requested byte amount :type retry_time: float :param retry_time: The length in time to wait to retry for the requested amount z<Request amount {} exceeded the amount available. Retry in {}N)� requested_amt� retry_time�format�super�__init__)�selfr r �msg� __class__s ��6/usr/lib/python3/dist-packages/s3transfer/bandwidth.pyr z!RequestExceededException.__init__ s8 �� � +���$���L�S�S� �:� �� ���� � )�__name__� __module__� __qualname__r � __classcell__)r s @r r r s �� � � r r c � � e Zd ZdZy)� RequestTokenzDA token to pass as an identifier when consuming from the LeakyBucketN)r r r �__doc__� r r r r ' s � �N�r r c � � e Zd Zd� Zd� Zy)� TimeUtilsc �* � t j � S )zgGet the current time back :rtype: float :returns: The current time in seconds )�time�r s r r zTimeUtils.time. s � � �y�y�{�r c �, � t j |� S )zwSleep for a designated time :type value: float :param value: The time to sleep for in seconds )r �sleep)r �values r r zTimeUtils.sleep6 s � � �z�z�%� � r N)r r r r r r r r r r - s � ��!r r c � � e Zd Zdd�Z dd�Zy)�BandwidthLimiterNc �D � || _ || _ |�t � | _ yy)a Limits bandwidth for shared S3 transfers :type leaky_bucket: LeakyBucket :param leaky_bucket: The leaky bucket to use limit bandwidth :type time_utils: TimeUtils :param time_utils: Time utility to use for interacting with time. N)� _leaky_bucket� _time_utilsr )r � leaky_bucket� time_utilss r r zBandwidthLimiter.__init__@ s) � � *���%��� � � (�{�D� � r c �n � t || j || j � }|s|j � |S )a� Wraps a fileobj in a bandwidth limited stream wrapper :type fileobj: file-like obj :param fileobj: The file-like obj to wrap :type transfer_coordinator: s3transfer.futures.TransferCoordinator param transfer_coordinator: The coordinator for the general transfer that the wrapped stream is a part of :type enabled: boolean :param enabled: Whether bandwidth limiting should be enabled to start )�BandwidthLimitedStreamr# r$ �disable_bandwidth_limiting)r �fileobj�transfer_coordinator�enabled�streams r �get_bandwith_limited_streamz,BandwidthLimiter.get_bandwith_limited_streamN s: � � (� �T�'�'�)=�t�?O�?O� �� � � -� -� /�� r �N)T)r r r r r. r r r r! r! ? s � � +� 6:�r r! c �\ � e Zd Z dd�Zd� Zd� Zd� Zd� Zd� Zd� Z dd �Z d � Z d � Z d � Z d � Zy)r( Nc � � || _ || _ || _ || _ |�t � | _ d| _ t � | _ d| _ || _ y)a[ Limits bandwidth for reads on a wrapped stream :type fileobj: file-like object :param fileobj: The file like object to wrap :type leaky_bucket: LeakyBucket :param leaky_bucket: The leaky bucket to use to throttle reads on the stream :type transfer_coordinator: s3transfer.futures.TransferCoordinator param transfer_coordinator: The coordinator for the general transfer that the wrapped stream is a part of :type time_utils: TimeUtils :param time_utils: The time utility to use for interacting with time NTr ) �_fileobjr# �_transfer_coordinatorr$ r �_bandwidth_limiting_enabledr �_request_token� _bytes_seen�_bytes_threshold)r r* r% r+ r& �bytes_thresholds r r z BandwidthLimitedStream.__init__f sU � �0 �� � )���%9��"�%��� � � (�{�D� �+/��(� *�n��� ��� /�� r c � � d| _ y)z0Enable bandwidth limiting on reads to the streamTN�r4 r s r �enable_bandwidth_limitingz0BandwidthLimitedStream.enable_bandwidth_limiting� s � �+/��(r c � � d| _ y)z1Disable bandwidth limiting on reads to the streamFNr: r s r r) z1BandwidthLimitedStream.disable_bandwidth_limiting� s � �+0��(r c �8 � | j s| j j |� S | xj |z c_ | j | j k r| j j |� S | j � | j j |� S )zhRead a specified amount Reads will only be throttled if bandwidth limiting is enabled. )r4 r2 �readr6 r7 � _consume_through_leaky_bucket)r �amounts r r> zBandwidthLimitedStream.read� s � � �/�/��=�=�%�%�f�-� -� ���F�"�� � � �d� 3� 3� 3��=�=�%�%�f�-� -� �*�*�,��}�}�!�!�&�)�)r c �r � | j j s9 | j j | j | j � d| _ y | j j �# t $ r/}| j j |j � Y d }~nd }~ww xY w| j j s���i)Nr ) r3 � exceptionr# �consumer6 r5 r r$ r r )r �es r r? z4BandwidthLimitedStream._consume_through_leaky_bucket� s� � � �,�,�6�6� 5��"�"�*�*��$�$�d�&9�&9�� $%�� �� �,�,�6�6� 6�� ,� 5�� � �&�&�q�|�|�4�4�� 5�� �,�,�6�6�s �7A& �& B �/%B�B c �$ � | j � y)z6Signal that data being read is being transferred to S3N)r; r s r �signal_transferringz*BandwidthLimitedStream.signal_transferring� s � � �&�&�(r c �$ � | j � y)z:Signal that data being read is not being transferred to S3N)r) r s r �signal_not_transferringz.BandwidthLimitedStream.signal_not_transferring� s � � �'�'�)r c �< � | j j ||� y r/ )r2 �seek)r �where�whences r rJ zBandwidthLimitedStream.seek� s � � � � ���5�&�)r c �6 � | j j � S r/ )r2 �tellr s r rN zBandwidthLimitedStream.tell� s � ��}�}�!�!�#�#r c � � | j r | j r| j � | j j � y r/ )r4 r6 r? r2 �closer s r rP z BandwidthLimitedStream.close� s1 � � � +� +��0@�0@� � .� .� 0� � � ��� r c � � | S r/ r r s r � __enter__z BandwidthLimitedStream.__enter__� s � �� r c �$ � | j � y r/ )rP )r �args�kwargss r �__exit__z BandwidthLimitedStream.__exit__� s � � � � � r )Ni )r )r r r r r; r) r> r? rF rH rJ rN rP rR rV r r r r( r( e sG � � �"� !0�F0�1�*�(7�$)�*�*�$� ��r r( c �8 � e Zd Z dd�Zd� Zd� Zd� Zd� Zd� Zy) � LeakyBucketNc �� � t |� | _ || _ |�t � | _ t j � | _ || _ |�t � | _ || _ |�t � | _ yy)a9 A leaky bucket abstraction to limit bandwidth consumption :type rate: int :type rate: The maximum rate to allow. This rate is in terms of bytes per second. :type time_utils: TimeUtils :param time_utils: The time utility to use for interacting with time :type rate_tracker: BandwidthRateTracker :param rate_tracker: Tracks bandwidth consumption :type consumption_scheduler: ConsumptionScheduler :param consumption_scheduler: Schedules consumption retries when necessary N) �float� _max_rater$ r � threading�Lock�_lock� _rate_tracker�BandwidthRateTracker�_consumption_scheduler�ConsumptionScheduler)r �max_rater& � rate_tracker�consumption_schedulers r r zLeakyBucket.__init__� sl � �. �x� ���%��� � � (�{�D� � �^�^�%�� � )��� � �!5�!7�D� �&;��#� � (�*>�*@�D� '� )r c �l � | j 5 | j j � }| j j |� r | j |||� cddd� S | j ||� r| j |||� n| j ||� cddd� S ddd� y# 1 sw Y yxY w)ac Consume an a requested amount :type amt: int :param amt: The amount of bytes to request to consume :type request_token: RequestToken :param request_token: The token associated to the consumption request that is used to identify the request. So if a RequestExceededException is raised the token should be used in subsequent retry consume() request. :raises RequestExceededException: If the consumption amount would exceed the maximum allocated bandwidth :rtype: int :returns: The amount consumed N) r^ r$ r ra � is_scheduled�,_release_requested_amt_for_scheduled_request� _projected_to_exceed_max_rate�!_raise_request_exceeded_exception�_release_requested_amt�r �amt� request_token�time_nows r rC zLeakyBucket.consume� s� � �$ �Z�Z� B��'�'�,�,�.�H��*�*�7�7� �F��H�H�� ��� B� B� �3�3�C��B��6�6�� ��� �2�2�3��A� B� B�� B� B� B�s �AB*� 7B*�*B3c �X � | j j ||� }|| j kD S r/ )r_ �get_projected_rater[ )r rm ro �projected_rates r ri z)LeakyBucket._projected_to_exceed_max_rate s) � � �+�+�>�>�s�H�M�� ����.�.r c �\ � | j j |� | j ||� S r/ )ra � process_scheduled_consumptionrk rl s r rh z8LeakyBucket._release_requested_amt_for_scheduled_request s0 � � �#�#�A�A� � � �*�*�3��9�9r c � � |t | j � z }| j j |||� }t ||�� �)N)r r )rZ r[ ra �schedule_consumptionr )r rm rn ro �allocated_timer s r rj z-LeakyBucket._raise_request_exceeded_exception% sF � � �u�T�^�^� 4�4���0�0�E�E� � �� � � '� �*� � r c �> � | j j ||� |S r/ )r_ �record_consumption_rate)r rm ro s r rk z"LeakyBucket._release_requested_amt. s � � ���2�2�3��A�� r )NNN) r r r r rC ri rh rj rk r r r rX rX � s. � � �� "� !A�F B�>/�:� �r rX c �$ � e Zd Zd� Zd� Zd� Zd� Zy)rb c � � i | _ d| _ y)z*Schedules when to consume a desired amountr N)� _tokens_to_scheduled_consumption� _total_waitr s r r z ConsumptionScheduler.__init__4 s � �02��-� ��r c � � || j v S )z�Indicates if a consumption request has been scheduled :type token: RequestToken :param token: The token associated to the consumption request that is used to identify the request. )r| )r �tokens r rg z!ConsumptionScheduler.is_scheduled9 s � � ��=�=�=�=r c �| � | xj |z c_ | j |d�| j |< | j S )a� Schedules a wait time to be able to consume an amount :type amt: int :param amt: The amount of bytes scheduled to be consumed :type token: RequestToken :param token: The token associated to the consumption request that is used to identify the request. :type time_to_consume: float :param time_to_consume: The desired time it should take for that specific request amount to be consumed in regardless of previously scheduled consumption requests :rtype: float :returns: The amount of time to wait for the specific request before actually consuming the specified amount. )� wait_duration�time_to_consume)r} r| )r rm r r� s r rv z)ConsumptionScheduler.schedule_consumptionB sA � �& ���O�+�� !� -� -� .�8 ��-�-�e�4� � � � r c �| � | j j |� }t | j |d z d� | _ y)z�Processes a scheduled consumption request that has completed :type token: RequestToken :param token: The token associated to the consumption request that is used to identify the request. r� r N)r| �pop�maxr} )r r �scheduled_retrys r rt z2ConsumptionScheduler.process_scheduled_consumption\ s= � � �?�?�C�C�E�J�� � � � ��/@� A� A�1� ��r N)r r r r rg rv rt r r r rb rb 3 s � � � >� �4 r rb c �< � e Zd Zdd�Zed� � Zd� Zd� Zd� Zd� Z y) r` c �. � || _ d| _ d| _ y)a� Tracks the rate of bandwidth consumption :type a: float :param a: The constant to use in calculating the exponentional moving average of the bandwidth rate. Specifically it is used in the following calculation: current_rate = alpha * new_rate + (1 - alpha) * current_rate This value of this constant should be between 0 and 1. N)�_alpha� _last_time� _current_rate)r �alphas r r z BandwidthRateTracker.__init__j s � � �� � ��� !��r c �4 � | j �y| j S )zmThe current transfer rate :rtype: float :returns: The current tracked transfer rate � )r� r� r s r � current_ratez!BandwidthRateTracker.current_ratez s � � �?�?� "���!�!�!r c �@ � | j �y| j ||� S )aZ Get the projected rate using a provided amount and time :type amt: int :param amt: The proposed amount to consume :type time_at_consumption: float :param time_at_consumption: The proposed time to consume at :rtype: float :returns: The consumption rate if that amt and time were consumed r� )r� �*_calculate_exponential_moving_average_rate�r rm �time_at_consumptions r rq z'BandwidthRateTracker.get_projected_rate� s* � � �?�?� "���>�>� �$� � r c �v � | j �|| _ d| _ y| j ||� | _ || _ y)a Record the consumption rate based off amount and time point :type amt: int :param amt: The amount that got consumed :type time_at_consumption: float :param time_at_consumption: The time at which the amount was consumed Nr� )r� r� r� r� s r ry z,BandwidthRateTracker.record_consumption_rate� sB � � �?�?� "� 1�D�O�!$�D� � � !� L� L� �$� ��� .��r c �J � || j z }|dk r t d� S ||z S )Nr �inf)r� rZ )r rm r� � time_deltas r �_calculate_ratez$BandwidthRateTracker._calculate_rate� s. � �(�4�?�?�:� � ��?� ��<� ��j�!�!r c �~ � | j ||� }| j |z d| j z | j z z S )N� )r� r� r� )r rm r� �new_rates r r� z?BandwidthRateTracker._calculate_exponential_moving_average_rate� s> � � �'�'��-@�A���{�{�X�%��T�[�[� �D�<N�<N�(N�N�Nr N)g�������?) r r r r �propertyr� rq ry r� r� r r r r` r` i s0 � �"� �"� �"� �$.�$"�Or r` ) r\ r � Exceptionr r r r! r( rX rb r` r r r �<module>r� sk �� � � �y� �, � � !� !�$#� #�Ln� n�bZ� Z�z3 � 3 �lNO� NOr