:j7 dZddlZddlZddlZddlZddlmZmZddlm Z m Z m Z ddl m Z e dZ Gdd ZGd d eZGd d eZGddeZGddeZGddeZddddefdfdedededede e edfdef d Z d+d!e e edfdededededef d"Zd#e d$e fd%Zd,d&ed$efd'Z d-ded(ededed)ef d*Zy).aY Unified Retry Framework - Comprehensive Retry Logic and Strategies This module provides a unified retry framework that combines: - Strategy Pattern: Object-oriented retry policies for pipeline stages - Decorator Pattern: Function decorators for automatic retry - Core Algorithms: Shared retry logic with exponential backoff and jitter Key Features: - Multiple retry strategies (Fixed, Exponential, Jitter, None) - Decorator support for sync/async functions - Smart error classification and retry decisions - Dynamic parameter extraction from function arguments - Unified configuration and behavior N)ABCabstractmethod)CallableTupleType) get_loggerretryc`eZdZdZededededefdZe d dede de d e d ede f d Z y ) RetryCorezFCore retry algorithms and utilities shared across all retry mechanismserrorattempt max_retriesreturnc||k\ryt|ttfryt|j }d|vsd|vryt|t t tfryy)a'Determine if an error should trigger a retry Args: error: Exception that occurred attempt: Current attempt number (0-based) max_retries: Maximum number of retries allowed Returns: bool: True if error should trigger retry FTz rate limitztoo many requests) isinstanceConnectionError TimeoutErrorstrlower ValueError TypeErrorKeyError)r rr error_strs 2/root/.openclaw/workspace/harvester/tools/retry.pyshould_retry_errorzRetryCore.should_retry_error'sc k ! eo|< =J$$& 9 $(;y(H ej)X> ? base_delay multiplier max_delayjitterc|||zz}t||}|r+d}tj| ||z}td||z}|S)aCalculate retry delay with exponential backoff and optional jitter Args: attempt: Current attempt number (0-based) base_delay: Base delay in seconds multiplier: Exponential multiplier max_delay: Maximum delay cap jitter: Whether to add random jitter Returns: float: Calculated delay in seconds 皙?)minrandomuniformmax)rrrr r!delay jitter_factor jitter_amounts rcalculate_delayzRetryCore.calculate_delayEsV"j'12E9% M"NNM>=IEQMU]23E rN)?@>@T) __name__ __module__ __qualname____doc__ staticmethod Exceptionintboolrfloatr+rrr r $s{P)cPT:pt"';@SXim rr cHeZdZdZedededefdZedede fdZ y) RetryPolicyz&Abstract base class for retry policiesrr rcy)zDetermine if task should be retried Args: attempt: Current attempt number (0-based) error: Exception that occurred Returns: bool: True if task should be retried Nr8selfrr s r should_retryzRetryPolicy.should_retryjs rcy)zGet delay before next retry attempt Args: attempt: Current attempt number (0-based) Returns: float: Delay in seconds Nr8r=rs r get_delayzRetryPolicy.get_delayws rN) r/r0r1r2rr5r4r6r>r7rAr8rrr:r:gsN0  C   d            rr:cFeZdZdZd dedefdZdededefdZ dedefd Z y ) FixedRetryzFixed delay retry policyrr(c ||_||_yN)rr()r=rr(s r__init__zFixedRetry.__init__s& rrr rcDtj|||jS)z'Check if should retry with fixed policyr rrr<s rr>zFixedRetry.should_retry++E7DrAr8rrrCrCsI"CENCN NdNrrCc NeZdZdZd dedededefdZdeded efd Z ded efd Z y )ExponentialBackoffz Exponential backoff retry policyrrr rc<||_||_||_||_yrE)rrr r)r=rrr rs rrFzExponentialBackoff.__init__s&$"$rrr rcDtj|||jS)z.Check if should retry with exponential backoffrHr<s rr>zExponentialBackoff.should_retryrIrcrtj||j|j|jdS)z#Calculate exponential backoff delayFr!r r+rrr r@s rrAzExponentialBackoff.get_delays.(($//4??TXTbTbkp(qqrN)rKr,r.r-rLr8rrrNrNs\*%C%%QV%kp% NCN NdNrrrrrNc TeZdZdZ d dededededef fd Zded efd ZxZS) JitterBackoffz8Exponential backoff with jitter to avoid thundering herdrrr rr)c8t|||||||_yrE)superrFr))r=rrr rr) __class__s rrFzJitterBackoff.__init__s  j)ZH*rrrcrtj||j|j|jdS)z/Calculate exponential backoff delay with jitterTrRrSr@s rrAzJitterBackoff.get_delays.(($//4??TXTbTbko(ppr)rKr,r.r-r#) r/r0r1r2r5r7rFrA __classcell__)rXs@rrUrUsbB" + + + +  +  +qqqrrUc4eZdZdZdededefdZdedefdZ y)NoRetryz"No retry policy - fail immediatelyrr rcy)z Never retryFr8r<s rr>zNoRetry.should_retrysrcy)zNo delay neededgr8r@s rrAzNoRetry.get_delaysrN) r/r0r1r2r5r4r6r>r7rAr8rrr\r\s2,C drr\rKr,r-Tr. max_attemptsr(backoff_factorr! exceptions.r c@dtdtffd }|S)aDecorator for automatic retry with exponential backoff using unified core Args: max_attempts: Maximum number of retry attempts delay: Initial delay between retries in seconds backoff_factor: Multiplier for delay after each failure jitter: Whether to add random jitter to delay exceptions: Tuple of exceptions to catch and retry max_delay: Maximum delay cap in seconds Returns: Decorated function with retry logic funcrctjfd}tjfd}tjr|S|S)Nc Xd}t D] }  |i|cS|#$r}|}| dz krUtj| }tj d|dzd d|d|dd t j |ntjd d |Yd}~d}~wwxYw NrzAttempt /z failed: z. Retrying in z.2fzs...zAll z attempts failed. Last error: )ranger r+loggerwarningtimesleepr argskwargslast_exceptionre actual_delayr`r(rarcr!r_r s r sync_wrapperz3with_retry..decorator..sync_wrappers!N . ] ]000 ]"! " ]%&N!11'0'@'@%Q_ajlr's &w{m1\N)A3N`amnq`rrvw <0 tL>9WXYWZ%[\ ]s B)A:B$$B)c Kd}t D]}  |i|d{cS|7 #$r}|}| dz kr^tj| }tj d|dzd d|d|dd t j |d{7ntjd d |Yd}~d}~wwxYwwrf)rhr r+rirjasynciorlr rms r async_wrapperz4with_retry..decorator..async_wrappers!N . ] ]!%t!6v!666 ]"! 7! ]%&N!11'0'@'@%Q_ajlr's &w{m1\N)A3N`amnq`rrvw&mmL999 tL>9WXYWZ%[\ ]sCC ,*,C,B>A!B9B!B94C9B>>C) functoolswrapsruiscoroutinefunction) rcrsrvr`r(rar!r_r s ` r decoratorzwith_retry..decoratorsa   ! !  !,   ! !  !.  & &t ,  r)r)r_r(r`r!rar rzs`````` r with_retryr{s$,3 3 X3 3 j rexception_typesc$t||||||S)aRetry decorator for specific exception types Args: exception_types: Tuple of exception types to retry on max_attempts: Maximum number of retry attempts delay: Initial delay between retries in seconds backoff_factor: Multiplier for delay after each failure jitter: Whether to add random jitter to delay max_delay: Maximum delay cap in seconds Returns: Decorated function with retry logic r_r(r`r!rar )r{)r|r_r(r`r!r s rretry_on_exceptionrs#* !%"  rrcrcBtjfd}|S)aSmart network retry decorator that extracts retry configuration from function parameters This decorator automatically extracts 'retries' and 'interval' parameters from the decorated function's arguments and applies dynamic retry logic accordingly. If these parameters are not provided, it falls back to sensible defaults. Args: func: Function to be decorated Returns: Decorated function with dynamic retry capability Example: @network_retry def http_get(url, retries=3, interval=1.0): # Function implementation pass # Uses custom retry configuration http_get("http://example.com", retries=5, interval=2.0) # Uses default configuration http_get("http://example.com") c||jdd}|jdd} |tdt|nd}|tdt|nd}t||d d ttfd }||i|S#tt f$r*d}d}t jdjdYawxYw) NretriesrKintervalr,rr#zInvalid retry parameters for z, using defaultsr-Tr.r~) getr'r5r7rrrirjr/r{rr)rnrorrretry_decoratorrcs rwrapperznetwork_retry..wrapperQs**Y*::j#. \.5.Ac!S\*qG4<4Hs3h0cH% '6  %t$d5f55#I& \GH NN:4==/IYZ [  \s2B6B;:B;)rwrx)rcrs` r network_retryr7s'4__T668 Nr policy_typec |j}|dk(r tdi|S|dk(r tdi|S|dk(r tdi|S|dk(r t St j d|dtdi|S)zFactory function to create retry policies Args: policy_type: Type of policy ('fixed', 'exponential', 'jitter', 'none') **kwargs: Policy-specific parameters Returns: RetryPolicy: Configured retry policy instance fixed exponentialr!nonezUnknown retry policy type: z, using exponential backoffr8)rrCrNrUr\rirj)rros rcreate_retry_policyrvs##%Kg#F##  %!+F++  &v&&  y4[MA\]^!+F++rrnetwork_errors_onlycP|r ttf}ntf}t||d|||S)afSmart retry decorator that uses intelligent error classification Args: max_attempts: Maximum retry attempts base_delay: Initial delay in seconds max_delay: Maximum delay cap jitter: Whether to add jitter network_errors_only: If True, only retry network-related errors Returns: Decorator function r-r~)rrr4r{)r_rr r!rras r smart_retryrs7&%|4 \ !  r)rKr,r-Tr.)r)rKr,r.TT)r2rurwr%rkabcrrtypingrrrrir r r:rCrNrUr\r4r5r7r6r{rrrrrr8rrrs  #(( G ;;F # :   rr$q&q& k $/8l KK KK  K d9os*+ K  K` 4 ?C/0      >77X7~,S,{,4 $     r