lunar_toolkit.downloader

PdsDownloader — parallel file retrieval from the PDS archive.

Progress = Callable[[str, int, int | None], None] module-attribute

progress(file name, bytes done, bytes total or None), called per chunk; raising aborts.

While the host is paused (throttling cooldown) it is called about once a second with done = -1.

PdsDownloader(cache_dir=DEFAULT_CACHE_DIR, max_workers=4)

Download and cache files from the PDS archive.

Parameters:
  • cache_dir (Path, default: DEFAULT_CACHE_DIR ) –

    Base directory for the local cache (default ~/.cache/lunar_toolkit).

  • max_workers (int, default: 4 ) –

    Thread-pool size for parallel downloads (default 8).

cache_path(url)

Return the local path that would be used to cache url.

ensure_cached(url, force=False, show_progress=True, expected_md5=None)

Return local path for url, downloading if not already cached.

download(product_rows, output_dir, url_col='PDS_URL', show_progress=True, force=False)

Download PDS image files in parallel.

Parameters:
  • product_rows (DataFrame) –

    Must contain a url_col column (e.g. from CumIndex.lookup()).

  • output_dir (Path) –

    Destination directory for .IMG files.

  • url_col (str, default: 'PDS_URL' ) –

    Column holding the download URL.

  • show_progress (bool, default: True ) –

    Show per-file progress bars.

  • force (bool, default: False ) –

    Download again even when <output_dir>/<file name> exists (by default it is reused; downloads land via .part files, so an existing file is complete).
Returns:
  • Copy of product_rows with LOCAL_PATH and DOWNLOAD_ERROR columns added. –

download_urls(urls, output_dir, show_progress=True, force=False)

Download a plain list of URLs in parallel.

Existing files are reused unless force (see download). Returns a dict mapping each URL to its local path (or None on error).

ChecksumError

Bases: IOError

Raised when a downloaded file does not match its expected MD5.

report_progress(callback)

Send the progress of every download_file in this thread / context to callback.

file_md5(path, chunk=8 << 20)

Return the hex MD5 digest of path (streamed, constant memory).

url_filename(url)

File name at the end of url's path (query string ignored).

download_file(url, dest, show_progress=True, expected_md5=None, resume=True, timeout=60.0, retries=8, retry_codes=RETRY_CODES, max_wait=None, progress=None)

Download url to dest, throttled per host and retried on throttling / errors.

Politeness: at most $LUNAR_MAX_PER_HOST (default 3) downloads run against one host at a time, and their requests start at least $LUNAR_MIN_INTERVAL seconds apart (default 1). Throttling replies (429, 503) pause every download to that host for the server's Retry-After (up to 2 h; else exponential back-off) and do not count as failures; they give up only after max_wait seconds in total (default $LUNAR_MAX_WAIT, else 6 h). Other retry_codes (5xx), connection errors and interrupted transfers are retried retries times with back-off. Each retry resumes the .part file; checksum mismatches are not retried. See _download_once.

progress (default: the report_progress callback, if any) gets the bytes done after each chunk; an exception it raises stops the download and keeps the .part file for a later resume.