Automating Metadata Extraction from Legacy Survey Files: Resolving Null GDAL Headers and ISO 19115 Compliance Gaps #

Legacy archaeological survey exports—particularly Total Station .csv dumps, ArcView 3.x .shp/.dbf pairs, and early-2000s CAD .dxf conversions—routinely strip spatial metadata during field-to-office transfers. When automated ingestion pipelines attempt to harvest provenance, coordinate reference system (CRS) tags, and collection dates, GDAL’s GetMetadata() frequently returns empty dictionaries. This silent failure breaks institutional compliance, halts repository indexing, and forces manual reconciliation across multi-year excavation campaigns. Within modern Heritage GIS Architecture & Fundamentals, deterministic metadata recovery is not optional; it is a prerequisite for audit-ready digital repositories. The following protocol provides exact CLI diagnostics, unambiguous Python/GDAL configurations, and spatial tolerance validations to restore automated extraction while aligning with ISO 19115 heritage preservation mandates.

Phase 1: Diagnostic Protocol for Silent Header Failures #

Before deploying automation, isolate the exact failure vector using a three-step diagnostic sequence. Execute these commands against a staging copy located at /srv/heritage/raw/legacy/.

1. Verify Raw Header Encoding #

Legacy .dbf attribute tables frequently default to CP850 or Windows-1252. When GDAL encounters mismatched code pages, it silently drops non-ASCII attribute names during metadata parsing.

ogrinfo -ro -so -al /srv/heritage/raw/legacy/2004_excavation.shp | grep -E "DBF Code Page|Encoding"

Expected Output: DBF Code Page: CP1252 Failure Indicator: Unknown or CP437 triggers attribute truncation. Remediate by exporting SHAPE_ENCODING=CP1252 prior to driver initialization.

2. Check Embedded vs. External CRS Tags #

Many legacy shapefiles lack .prj sidecars but embed projection strings in custom *.xml manifests or header comments. Validate WKT presence and truncation:

gdalinfo -json /srv/heritage/raw/legacy/2004_orthophoto.tif | jq -r '.coordinateSystem.wkt // "NULL"'

Tolerance Threshold: If the WKT string is truncated or returns NULL, the driver defaults to WGS 84 (EPSG:4326) with a ±0.000001° geographic tolerance, which is unacceptable for site-scale UTM projections.

3. Audit Metadata Dictionary Population #

In Python, verify whether the driver bypasses metadata extraction due to missing MD blocks or driver-specific header corruption:

from osgeo import gdal
gdal.UseExceptions()
ds = gdal.OpenEx("/srv/heritage/raw/legacy/2004_excavation.shp", gdal.OF_VECTOR)
print("Native:", ds.GetMetadata())
print("Image Structure:", ds.GetMetadata("IMAGE_STRUCTURE"))

Failure Indicator: Both return {}. This confirms the OGR driver is operating in legacy fallback mode and requires explicit heuristic harvesting.

Phase 2: Deterministic Extraction Pipeline #

The following Python routine forces deterministic extraction, applies heuristic fallbacks, and maps recovered fields to ISO 19115-compliant structures. It assumes a POSIX-compliant environment with Python 3.10+, GDAL 3.6+, and pyproj 3.5+.

import os
import re
import json
from pathlib import Path
from osgeo import gdal, ogr
import pyproj
from lxml import etree

# 1. Enforce deterministic environment variables
os.environ["SHAPE_ENCODING"] = "CP1252"
os.environ["GDAL_FILENAME_IS_UTF8"] = "NO"
os.environ["OGR_ENABLE_PARTIAL_REPROJECTION"] = "YES"
gdal.UseExceptions()

def extract_legacy_metadata(input_path: str) -> dict:
    """Extracts and normalizes metadata from legacy survey files."""
    path = Path(input_path)
    if not path.exists():
        raise FileNotFoundError(f"Target path does not exist: {input_path}")
        
    ds = gdal.OpenEx(str(path), gdal.OF_VECTOR | gdal.OF_READONLY)
    if ds is None:
        raise RuntimeError(f"GDAL failed to initialize driver for {input_path}")
        
    layer = ds.GetLayer()
    meta = {}
    
    # 2. Native extraction
    native_meta = ds.GetMetadata()
    meta.update(native_meta)
    
    # 3. Heuristic fallback for null dictionaries
    if not meta:
        # Harvest from sidecar XML if present
        xml_sidecar = path.with_suffix(".xml")
        if xml_sidecar.exists():
            tree = etree.parse(str(xml_sidecar))
            for elem in tree.iter():
                if "date" in elem.tag.lower() or "proj" in elem.tag.lower():
                    meta[f"sidecar_{elem.tag}"] = elem.text.strip()
                    
        # Harvest from DBF field names (common legacy practice)
        for field_def in layer.schema:
            name = field_def.GetName()
            if re.match(r"^(date|survey|crs|proj|epsg)", name, re.IGNORECASE):
                meta[f"field_{name}"] = True
                
    # 4. CRS validation and normalization
    crs_wkt = layer.GetSpatialRef().ExportToWkt()
    if crs_wkt:
        crs_obj = pyproj.CRS.from_wkt(crs_wkt)
        meta["iso_crs_auth"] = crs_obj.to_authority() if crs_obj.is_epsg_code() else "UNKNOWN"
        meta["iso_crs_wkt"] = crs_wkt
        
    # 5. Bounding box extraction
    extent = layer.GetExtent()
    meta["bbox"] = {
        "minx": round(extent[0], 6),
        "miny": round(extent[1], 6),
        "maxx": round(extent[2], 6),
        "maxy": round(extent[3], 6)
    }
    
    return meta

# Execution example
if __name__ == "__main__":
    result = extract_legacy_metadata("/srv/heritage/raw/legacy/2004_excavation.shp")
    print(json.dumps(result, indent=2))

Phase 3: Spatial Validation & ISO 19115 Mapping #

Automated extraction must be followed by strict spatial validation before repository ingestion. Coordinate precision tolerances are non-negotiable for heritage datasets:

After validation, map extracted fields directly to ISO 19115-1:2014 elements. The Metadata Standards for Archaeological Data mandate explicit lineage, temporal coverage, and spatial reference blocks. Use the following mapping schema for pipeline output:

• meta["iso_crs_auth"] → MD_ReferenceSystem/identifier
• meta["bbox"] → EX_GeographicBoundingBox
• meta["sidecar_date"] or meta["field_date"] → CI_Date (creation/modification)
• native_meta.get("DESCRIPTION") → MD_DataIdentification/abstract

Refer to the official ISO 19115 Geographic Information — Metadata specification for exact XML schema validation rules. When implementing cross-platform GIS interoperability testing, validate the generated metadata against QGIS 3.34+ and ArcGIS Pro 3.1+ parsers to ensure bidirectional compatibility.

Phase 4: Pipeline Integration & Preservation Workflow #

Embedding this extraction routine into automated ingestion requires strict project scoping and data governance controls. Prior to execution, define a staging directory (/srv/heritage/staging/) with immutable read permissions. All legacy files must be copied via rsync -a --checksum to prevent timestamp drift during validation.

When integrating with existing workflows, such as Setting Up QGIS for Archaeological Surveys (note: contextual reference), ensure the pipeline outputs a standardized JSON-LD manifest alongside the spatial dataset. This manifest should be version-controlled and linked to the primary dataset via persistent identifiers (DOIs or ARKs).

For long-term digital preservation, archive the raw legacy files alongside the extracted ISO 19115 metadata in a WARC or BagIt container. This guarantees that future migrations can reconstruct provenance even if GDAL driver behavior changes. Always document CRS selection decisions during initial ingestion, particularly when reconciling legacy local grids with modern national datums, as outlined in standard CRS Selection for Heritage Sites guidelines.

By enforcing deterministic environment variables, explicit spatial tolerances, and ISO-compliant field mapping, heritage teams can eliminate silent metadata failures and maintain audit-ready repositories across multi-decade excavation campaigns.