lunar_toolkit.lroc.cumindex

LROC PDS CUMINDEX as a local DuckDB / Parquet database.

CumIndex(dataset='edr', data_dir=None, cache_dir=DEFAULT_CACHE_DIR, release=None)

Query the LROC PDS CUMINDEX via DuckDB on a local Parquet database.

Parameters:
  • dataset (str, default: 'edr' ) –

    Which PDS dataset to use: "edr" (default), "cdr", or "rdr".

  • data_dir (Path, default: None ) –

    Directory containing cumindex_<dataset>.parquet. Defaults to LUNAR_DATA_DIR, else the user data dir.

  • cache_dir (Path, default: DEFAULT_CACHE_DIR ) –

    Where raw CUMINDEX/INDEX .TAB / .LBL downloads are cached. Defaults to ~/.cache/lunar_toolkit/.

  • release (str, default: None ) –

    PDS volume to read labels / index tables from (e.g. "LROLRC_0066C"). Defaults to the release recorded in the build sidecar, else the fallback constant. build_db() auto-discovers the newest volume.

release property

Volume used for label / index URLs (no trailing slash).

parquet_path property

Path to the Parquet database for this dataset.

meta_path property

Build-metadata sidecar next to the Parquet (*.meta.json).

schema property

Column definitions from CUMINDEX.LBL (lazy, downloads to cache).

meta property

Top-level scalar fields from CUMINDEX.LBL (lazy).

volume_index_urls(volume)

(INDEX.LBL, INDEX.TAB) URLs for a single PDS volume.

build_info()

Return the build sidecar as a dict, or None if absent.

discover_volumes()

List this dataset's PDS volumes (sorted oldest → newest).

Uses the S3 ListObjectsV2 listing exposed by the PDS store; falls back to probing INDEX/INDEX.LBL of successive volume numbers.

latest_release()

Newest volume on the server (fallback: the constant release).

build_db(force=False, release=None, volume_only=False, memory_limit='4GB', threads=None, wkt=False, on_stage=None)

Download the PDS index table and build the Parquet database.

Parameters:
  • force (bool, default: False ) –

    Rebuild even if the Parquet already exists.

  • release (str, default: None ) –

    Volume to build from (e.g. "LROLRC_0066C"). Default: newest volume discovered on the server, else the fallback constant.

  • volume_only (bool, default: False ) –

    Build from that volume's own INDEX/INDEX.TAB (one release, ~20 k rows) instead of the cumulative CUMINDEX.TAB (~3.8 GB).

  • memory_limit (str, default: '4GB' ) –

    DuckDB memory_limit; sorts spill to a temp dir under the cache.

  • threads (int, default: None ) –

    DuckDB worker threads (default: all cores).

  • wkt (bool, default: False ) –

    Also store a FP_WKT footprint polygon column.

  • on_stage (callable, default: None ) –

    Called with "download" and "build" as those stages start.
Returns:
  • Path to the written Parquet file. –

update_db(volumes=None, memory_limit='4GB', threads=None, on_stage=None)

Append per-volume INDEX.TAB files newer than the built release.

Parameters:
  • volumes (iterable of str, default: None ) –

    Explicit volumes to add. Default: every discovered volume newer than the sidecar's release (or max(VOLUME_ID) in the Parquet when no sidecar exists).

  • on_stage (callable, default: None ) –

    As in build_db.
Returns:
  • The list of volumes that were appended (empty if already up to date). –
  • Rows are de-duplicated on ``PRODUCT_ID`` (newest volume wins). –

query(sql)

Run arbitrary DuckDB SQL against the Parquet database.

Use the alias cumindex as the table name:

ci.query("SELECT PRODUCT_ID, EMISSION_ANGLE FROM cumindex LIMIT 10")

lookup(product_ids, columns=None)

Return metadata rows for the given product IDs.

Queries the Parquet via DuckDB — no full dataset loaded into RAM.

Parameters:
  • product_ids (list[str]) –

    e.g. ["M1524360231LE", "M1512659523RE"]

  • columns (list[str], default: None ) –

    Subset of columns to return. PRODUCT_ID and FILE_SPECIFICATION_NAME are always included.

search(lat=None, lon=None, point=None, radius_km=0.0, emission_angle=None, incidence_angle=None, phase_angle=None, nac_only=False, columns=None, wkt=False, limit=None, aoi=None)

Return images whose footprint intersects a region.

Region: either a lat/lon box (lat=(min, max), lon=(west, east); west > east crosses the 0/360 seam, longitudes may be given in −180..180 or 0..360, omitted lon = all longitudes), a circle (point=(lat, lon), radius_km) or an area of interest (aoi=: GeoJSON / WKT path, WKT string or Aoi).

Matching is bounding-box based and conservative (may include a few false positives, never drops a true hit for convex-ish footprints): the image's lat range and seam-aware longitude arc must overlap the query's, and for images poleward of ±60° their polar stereographic x/y box must overlap the query's projected box. Footprints that enclose a pole are treated as spanning all longitudes. With aoi the bbox matches are then refined exactly: each corner footprint polygon must intersect the AOI polygon (in polar stereographic when the AOI lies poleward of ±60°), and limit applies after that.

Example

::

df = ci.search(lat=(-90, -85), nac_only=True, incidence_angle=(80, 95))
df = ci.search(lat=(-10, 10), lon=(355, 5))         # across 0°
df = ci.search(point=(-89.67, 129.8), radius_km=15) # Shackleton
df = ci.search(aoi="aoi.geojson")                   # QuickMap polygon

get_images(product_ids)

Look up product IDs and return a NacImageCollection.

Single DuckDB query; each NacImage is pre-populated with its metadata row so subsequent property accesses require zero additional DB calls.

Parameters:
  • product_ids (list[str]) –

    e.g. ["M1524360231LE", "M1512659523RE"]
Returns:
  • NacImageCollection –

    Contains only the IDs that were found in the index, in input order.

load(nrows=None)

Load the full index into memory as a DataFrame.

For most workflows prefer lookup() or search(). Pass nrows for a quick preview that needs no Parquet: only the label and the first nrows records of the TAB are fetched (HTTP Range request), then parsed with the same DuckDB extraction as build_db().

to_observations(df)

Convert query results to the common multi-instrument schema.

build_parquet(sources, parquet_path, base_parquet=None, temp_dir=None, memory_limit='4GB', threads=None, wkt=False, expected_rows=None)

Build a sorted, ZSTD-compressed Parquet from fixed-width PDS TAB file(s).

Parameters:
  • sources (list of (tab_path, lbl_columns)) –

    Each TAB is parsed with its own label layout (record lengths differ between volumes) and the results are combined UNION ALL BY NAME.

  • base_parquet (Path, default: None ) –

    Existing database to merge with (incremental update). Rows are then de-duplicated on PRODUCT_ID, keeping the newest VOLUME_ID.
Returns:
  • int –

    Number of rows written.
Notes

Everything happens in one COPY (...) TO statement; DuckDB spills the sort to temp_dir.

column_sql(col)

SQL expression extracting one LBL column from the line VARCHAR.

Bytes → substr(line, START_BYTE, BYTES), trimmed of blanks and quotes, typed by DATA_TYPE (falling back to name hints) with TRY_CAST; NOT_APPLICABLE_CONSTANT / MISSING_CONSTANT → NULL.

footprint_sql(names, wkt=False)

Derived FP_* column expressions over an extracted-row relation.

  • FP_MIN_LAT / FP_MAX_LAT — corner (and centre) latitude range, extended to ±90 when the footprint encloses a pole.
  • FP_LON_WEST / FP_LON_EAST / FP_LON_SPAN — the shortest longitude arc covering all corners, in [0, 360); WEST > EAST means the arc crosses the 0/360 seam; SPAN = 360 means all longitudes.
  • FP_CONTAINS_POLE — winding number of the corner ring ≠ 0.
  • FP_PS_* — polar stereographic bbox (m, R = 1737.4 km) for images whose centre is poleward of ±60° (FP_PS_HEMI = 'S' / 'N').

footprint_wkt_sql(names, west_expr='FP_LON_WEST')

SQL for a POLYGON((lon lat, ...)) WKT of the corner ring.

Longitudes are unwrapped relative to the footprint's western edge so the ring never jumps at the seam (values may exceed 360). For pole-enclosing footprints the lat/lon ring is only indicative — use the FP_PS_* box.

region_sql(region)

WHERE clause matching FP_* columns against a query_region.

parse_s3_listing(xml_text)

Return (common_prefixes, next_continuation_token) from ListObjectsV2 XML.

list_store_volumes(base_url, volume_re, timeout=30.0)

List volume directories under a dataset on the PDS S3 store.

probe_volumes(base_url, start, max_numbers=12, timeout=15.0)

Probe <vol>/INDEX/INDEX.LBL for volumes at/after start.

Volume names look like LROLRC_0066C (number + optional A/B/C… part letter). Stops after two consecutive numbers without any hit.

clean_stale_build_files(directory)

Delete leftovers of interrupted builds (*.build.duckdb[.wal], *.tmp).

parse_lbl_columns(lbl_content)

Parse OBJECT = COLUMN blocks of a PDS3 label (pure Python).

Keys are lower-cased LBL keywords; column_number, start_byte and bytes are ints. Sorted by column_number.