Synchronized Spider¶
This abstract class can be extended to produce new plugins that require a system-wide configuration synchronization.
pathspider.sync¶
-
class
pathspider.sync.
SemaphoreN
(value)[source]¶ An extension to the standard library’s BoundedSemaphore that provides functions to handle n tokens at once.
-
acquire_n
(value=1, blocking=True, timeout=None)[source]¶ Acquire
value
number of tokens at once.The parameters
blocking
andtimeout
have the same semantics asBoundedSemaphore
.- Returns
The same value as the last call to BoundedSemaphore’s
acquire()
ifacquire()
were calledvalue
times instead of the call to this method.
-
-
class
pathspider.sync.
SynchronizedSpider
(worker_count, libtrace_uri, args, server_mode=False)[source]¶ -
configurations
= []¶
-
configurator
()[source]¶ Thread which synchronizes on a set of semaphores and alternates between two system states.
-
worker
(worker_number)[source]¶ This function provides the logic for configuration-synchronized worker threads.
- Parameters
worker_number (int) – The unique number of the worker.
The workers operate as continuous loops:
Fetch next job from the job queue
Perform pre-connection operations
Acquire a lock for “config_zero”
Perform the “config_zero” connection
Release “config_zero”
Acquire a lock for “config_one”
Perform the “config_one” connection
Release “config_one”
Perform post-connection operations for config_zero and pass the result to the merger
Perform post-connection operations for config_one and pass the result to the merger
Do it all again
If the job fetched is the SHUTDOWN_SENTINEL, then the worker will terminate as this indicates that all the jobs have now been processed.
-