.
This commit is contained in:
commit
55385f8857
69 changed files with 1125 additions and 0 deletions
10
fetcher/README
Normal file
10
fetcher/README
Normal file
|
|
@ -0,0 +1,10 @@
|
|||
idea:
|
||||
given a list of URL-to-path pairs, it downloads them in
|
||||
parallel to the specified path.
|
||||
|
||||
implementation:
|
||||
These pairs are added to a queue(FIFO) along with an
|
||||
attempt counter. Multiple worker processes are spawned
|
||||
that each handle an entry in a queue.
|
||||
|
||||
workers write to an error pipe and to a out pipe.
|
||||
Loading…
Add table
Add a link
Reference in a new issue