:jcsUdZddlZddlZddlZddlZddlZddlZddlZddlZ ddl m Z m Z m Z mZmZmZddlZddlmZddlmZddlmZmZmZedZej6adt8_d adJd ed ej6fd Z d ejBjDd e#fdZ$d ejBjDd efdZ%d eed dfdZ&dKdedede'de d ejPf dZ)ddl*m+Z+m,Z,ddl-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7m8Z8ddl9m:Z:m;Z;ddlmZ>ddl?m@Z@ddlAmBZBddlCmDZDddlEmFZFddlmGZGGd d!ZHdaIeeHeJd"<d#e eeeed e#fd?ZXd>ed/ed e#fd@ZY dPd/ed0ed1e#d:eUd5e#d7ee eeegdfd eeeeffdAZZeGgd < dQdBeded*e#dCedDedEedFeed eefdGZ[eGgd <dFedHed eefdIZ\y)RzS HTTP client utilities and GitHub-specific search functions for the search engine. N)AnyCallableDictListOptionalTuple)Service) get_logger) encoding_urlisblanktrimsearchFproxyreturnc|tj}d|_|r|jj ||d|S)NF)httphttps)requestsSession trust_envproxiesupdate)rsessions 4/root/.openclaw/workspace/harvester/search/client.py _new_sessionrs7 GG >? Nerrorc<t|dd}| |jSdS)Nresponser)getattr status_code)rr s rhttp_error_statusr#&s&uj$/H#+#78  >Q>rct|dd}| t|S |jjdj d}t |dkDr|dd}|xs|jxs t|S#t $rd}Y*wxYw)Nr  i,r)r!strtext removeprefix removesuffixlen Exceptionreason)rr messages rhttp_error_messager.+suj$/H5z--,,T2??E w<# dsmG  3hoo 3U3 s=A66 BBc t|}|s"datatj dyt jj|}|jj}|dvr td|js td |jt|a|atj d|d |jd |jxsdy#t$r}td||d}~wwxYw) zIConfigure the process-wide requests session used by search HTTP requests.rzHTTP proxy disabledN>rrsocks5z0proxy scheme must be one of: http, https, socks5zproxy must include a hostzinvalid proxy port: zHTTP proxy enabled: z://:)r _HTTP_PROXYr _HTTP_SESSIONloggerinfourllibparseurlparseschemelower ValueErrorhostnameport)rparsedr9es r set_proxyr@:s KE  $  )* \\ " "5 )F ]] "F 00KLL ??455< !'MK KK&vhc&//1B!FKKDUSUCVWX </s34!;?rc.ts tStS)zGet GitHub client instance)rrbrGrrget_github_clientrs ~ rc vtrtjrtjj}i}|jj D]c\}}|j |j |j|j|j|j|j|jd||<e|SiS)zGet rate limiter statistics)raterttokens utilizationconsecutive_successconsecutive_failuresadaptive original_rate) rrc get_statsservicesitemsrrtrrrrrr)statsrrm bucket_statss rget_github_statsrs.00&&002%*^^%9%9%;  !G\$))%++&--+77'3'G'G(4(I(I(11!-!;!; F7O  IrcPtrtjsytjj}|jj D]T\}}t j d|d|jdd|jdd|jd|jd  Vy) z#Log current rate limiter statisticsNzRate limiter [z]: rate=rpz /s, tokens=z.1f/z, utilization=z.1%) rrcrrrr4r5rrrtr)rrmrs rlog_github_statsrs !7!7  " " , , .E!&!5!5!7  WIXl.?.?-DE"))#.a 0B0B/CD'33C8 :  rrrrrct|r tdd|xstj}t d|} t |}|r@t |tr0tjj|}d|vrdnd}|||z }ttd|||d 5} | j} | j} | d k7rtd | d | | j!d cdddS#t"$rI t%j&| j!d cYcdddS#t($r tdwxYwwxYw#1swYyxYw#t*j,j.$r} t1| } t3| }| dk(rt5d| d| dk(rt7d| d|| dvrtd| d| dk\rt5d| d|td | d|d} ~ wt*j,j8$r} t;d| d} ~ wt*j,j<$r} t5d| d} ~ wt($r<} dt?| jAvrt;d| td| d} ~ wwxYw) aHTTP GET request with configurable retry handling Args: url: URL to request headers: HTTP headers params: URL parameters retries: Number of retry attempts (default: 3, minimum: 1) interval: Initial delay between retries in seconds (default: 1.0, minimum: 0.1) timeout: Request timeout in seconds Returns: str: Response content Raises: ValidationError: For invalid input NetworkError: For network-related issues FileNotFoundError: For access resource not exists ConnectionError: For connection failures (will be retried) TimeoutError: For timeout errors (will be retried) Note: The @network_retry decorator automatically extracts retries and interval parameters to configure retry behavior dynamically. Retry logic uses exponential backoff with jitter for optimal performance. zURL cannot be emptyrB)fieldrF?&GET)rrChttp_connectionzHTTP z error for URL: zutf-8Nz!Failed to decode response contentizRate limit exceeded (HTTP )izFile not found (HTTP z), url: )rzAuthentication failed (HTTP izServer error (HTTP z): z error: zRequest timeout: zRequest error: rCzUnexpected error: )!r rYrOcopyrIr isinstancedictr6r7 urlencoder^rHcontentr"rXdecodeUnicodeDecodeErrorgzip decompressr+r exceptions HTTPErrorr#r.ConnectionErrorFileNotFoundErrorTimeout TimeoutErrorRequestExceptionr&r:)rBrrrrrC encoded_urldata separatorr rr"r?coder,s rrrsHs|35AA/--/G!WoG<9"3' j.<<))&1D"k1sI i[/ /K  E; IK\  L &&G"..Kc!"U;-7Gu#MNN L~~g. L L& LL??73::7CC L L !L&'JKKL L L L&    ( (? ##A& 3;!$>tfA"FG G S[#&;D6#$OP P Z !=dV1EF F S[!$7vS"IJ JtfHVH=> >    & &4.qc233    / /5s344 9 A  &!21#67 7!3A378 89sA&E$/EC.$ E. E8#D'EE E'D<<EEE E EJ ,A:G&& J H J 4I J 7JJ modelc ddtdtdtddffd }tt|c}stj dy t tstj d y td k(rd d <|rt |ts&|stj dy |ddtdgd}tj|jd}td|}td|}d\}} } | |kr7 td||5} | j}| j } ddd|| fS|| fS#1swYnxYwn#t"j$j&$r} t)| }|dk7r t+| } | j-dr| j/ds-| j0| j0j2n t| } n#t4$rt| } YnwxYw||| d|t6vrYd} ~ || fSYd} ~ n/d} ~ wt4$r ||t9j:dYnwxYw| dz } t=j>t@| |kr_*)z'Make chat API request with retry logic.Frr-rrNc|ddd|d|}|rtj|ytj|y)Nz[chat] failed to request URL: z , headers: z, status code: z , message: )r4rr)rr-rr'rrBs routputzchat..output}sA/uKyX\W]]hiphqr  LL  LL rz[chat] url cannot be empty)Nz[chat] headers must be a dictrapplication/jsonz content-typez[chat] model cannot be emptyuser)roler)rstreammessagesutf8rF)rNrPOST)rrrCr{})rr-rT)F)!rr&rr r4rrrr*rPjsondumpsencoderIrHr"r'rrrr#r. startswithendswithr r,r+rU traceback format_excrurvrM) rBrrrrrCrpayloadrr-attemptr r?s `` rchatrxs[ S3tcDKJC  12 gt $ 45 W "4 FD1 LL7 8"(5EFG  jj ''/G!WoG!WoG)D'7 G  J7GWU Ya++"-- 8 =4=9   "",, $Q'Ds{%03G#--c2':J:J3:O78zz7M!**"3"3SVWXSY %!!fG%D'?++ =, J i&:&:&Sign in to GitHub · GitHubflagszO[GithubCrawl] Session has expired, please provide a valid session and try again) r r\rrrerIr4r)rrrrBrclientrs rsearch_github_webrs qyGG$ /v]5' JCQ'$&!'+ G FjjS'j2G yy>rttT fg NrrFtoken peer_pagec<t|s t|rgStt|dttd|}}d|d|d|}dd|dd}t }|j ||t t }t|rgS tj|j d g}t} |D]F} | rt| tk7r| j d d } t| r6| j| Ht| S#t$rgcYSwxYw) zRate limit: 10RPM.rF%https://api.github.com/search/code?q="&sort=indexed&order=desc&per_page=&page=application/vnd.github+jsonBearer 2022-11-28r AuthorizationzX-GitHub-Api-VersionrBrrrCrhtml_urlrr minrIrKrrrQrSrloadssettyperaddlistr+) rrrrrBrrrrlinksitemlinks rsearch_github_apirs"u~ #i+-ABC4LtI 1%8Z[dZeeklpkq rC/"5'* ,G  FjjS'G ::g D 88M1 %%rrct|s try tjj} |t |d}t |}|stjd|dtSdvr'tfdd Drjd d }ntjj}d |}td dd|ddd|d}tjt!j t"zt%}|j'||d}|r|t)j*|} | j'ddsU| j'dd} | j'dd} tj-d| d| d|| dkDr| St/|St/|S#t$r}YwxYw#t$r,} tj1d|d| dtcYd} ~ Sd} ~ wwxYw)z Get total count for web search using GitHub's blackbird_count API. Performs a single search and then queries the count API. rNrFrrrz*[search] initial search failed for query: z, using conservative estimate%c3&K|]}|v ywNrG).0crs r z%estimate_web_total..sXqU Xs)z%2Fz%5Bz%5Dz%7Bz%7D +z=https://github.com/search/blackbird_count?saved_searches=^&q=rzgzip, deflate, brzhttps://github.com/search?q=z ^&type=codeXMLHttpRequestr)rrzAccept-EncodingrzX-Requested-Withrr failedTcountmoderoz [search] got z results, mode: z , query: z&[search] estimation failed for query: z , error: )r r6r7 unquote_plusr+rr r4warningrLanyreplacerr\rurvrandomrTrrrrr5extract_count_from_pager) rrrr-encodedrBrrr rr.r/r?s ` rr r s w75>,,++E21$ ?'eW1MGw- NNGyPmn o' ' %C93C>9C>cjtjjt|}|sgdfS|rt ||||}|dfSt |||}t |rgdfS d} tj| |tj} | rtt| ng} t} | D]} | jd| t| }|rt|tr |r |||||fS#t$r"}t j#d|Yd}~*d}~wwxYw#t$rgdfcYSwxYw) zR Search code with unified interface. Returns: (results_list, content) r)rrrrr#r rrr N)r6r7rr rrr rr rrrrrrr+r4r)rrrrrrkeywordrrrrrrrr?s r search_coder?s:ll%%d5k2G 2v #'tW`a{gwTJGw2v 9E7"$$7$*tCK  2C II*3%0 1 2u+  8X67 ?'* ? 9!=>> ? 2v s="BD"' C40D"4 D=DD"DD"" D21D2 key_patternaddress_patternendpoint_pattern model_patternr'c t|tst|trt|tsgS|r|}nt||t}|sgSt |}t ||}|sgSt |}t ||} |r| sgS| s| j dt |}t ||} |r| sgS| s| j dt |}t ||} |r| sgS| s| j dt} tj|| | | D]%\} }}}| j t||| |'| S)aSExtract API keys and related information from URLs or text content Args: key_pattern: Regex pattern to match API keys url: URL to fetch content from (if text not provided) retries: Number of retry attempts for HTTP requests address_pattern: Regex pattern to match service addresses endpoint_pattern: Regex pattern to match endpoints model_pattern: Regex pattern to match model names text: Text content to search (if provided, url is ignored) Returns: List[Service]: List of Service objects with extracted information )rBrr)r'rr)addressendpointkeyr) rr&rrNr extractappendr itertoolsproductr )r@rBrrArBrCr'rkeys addresses endpointsmodels candidatesrGrErFrs rcollectrQsW0 sC D#)>zR]_bGc  sG>TU  {#K { 3D  ?+OWO$q')A3>Z[ \  Dt*C #   ;r)r)r)rN)NNrg?r)rNr)rFNr&)rrrrrN)]rrrJrr4rrur urllib.parser6typingrrrrrrr core.modelsr tools.loggerr tools.utilsr r r r4rr3rr2r&rrrrr#r.r@rResponserHconstant.searchrKrLconstant.systemrMrNrOrPrQrRrSrTrUrVrWcore.exceptionsrXrYrZ core.typesr[tools.coordinatorr\tools.ratelimitr]tools.resourcesr^ tools.retryr_r`rbr__annotations__rrrrrrrrrrrrr!r r5r?rQrHrGrrrhs  ==#33 H    "   X%5%5?X00::?s? 4h11;; 4 4YXc]YtY:CcE#(J[J[G    :'$,'+%)^^D*.&-@tC$89@d@<$sDe$445*  #! f9 f9 d^f9 TNf9 f9  f9  f9 f9f9ToqE EE$'E6>tnEVYEhkE 38_EPS3cc,;$c>$C>$(3->$SV>$B$S$$$N<@ , , , , ,  , xcC 0$ 678 , 49c>,^"8FF FF F  F  F 3-F ']F9FR"8#cd3i9r