Versions Compared

Old Version 9

changes.mady.by.user Dustin Lo

Saved on

compared with

New Version Current

changes.mady.by.user Dustin Lo

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Gerald Manipon edited this page on May 16 · 7 revisions

(blue star) Confidence Level TBD  This article has not been reviewed for accuracy, timeliness, or completeness. Check that this information is valid before acting on it.

...

  • the parallelized function utilizes multiprocessing (through celery’s billiard library)

  • CPUs used will depend on how many available CPUs on the worker and how many datasets to publish

    • Code Block
      languagepy
      num_procs = min(max(cpu_count() - 2, 1), len(datasets_list))

    • parallelization will scale up depending on how many CPUs on the worker; more CPUs → faster localization/publish

to use the parallelized localization/publish the PGE’s job spec would have to be changed:

...