A graph pipeline from first principles

A production graph pipeline is a sequence of explicit, replayable transformations rather than a direct loader into a graph database. The minimum stages are source capture, immutable raw storage, structural parsing, semantic mapping, preliminary validation, identity resolution, enrichment or reasoning, release validation, and atomic publication. Each stage accepts versioned inputs and emits artifacts, metrics, and provenance. This architecture makes failures isolatable and releases reproducible.

Capture source material before interpretation. Store bytes or records with a content digest, source locator, retrieval time, access classification, and parser hint. A digest supports deduplication and proves which input was processed. Do not use a mutable URL as the only evidence identifier. Privacy and licensing policy may require encrypted storage, minimization, expiration, or storing a cryptographic reference rather than full content.

Parsing converts bytes into source-native records without domain claims. Preserve page, row, field, and character offsets so mapped assertions can cite exact evidence. Parsing errors belong to a dead-letter or review stream with machine-readable codes. Silently dropping malformed rows creates unknown graph gaps that no later validation can diagnose.

Mapping turns source fields into IRIs, typed literals, and vocabulary terms. Put mappings in reviewed code or declarative rules under version control. A mapper should be deterministic for the same source artifact and configuration. Generate assertion identifiers from stable inputs only if collision and correction behavior are understood; otherwise mint IDs and maintain an idempotency key. Language tags, time zones, units, and enumerated codes need explicit conversions.

Validate twice. Pre-resolution shapes catch missing identifiers, invalid datatypes, and mapping defects on source records. Post-resolution shapes check canonical relationships, cardinalities, provenance, and release invariants. Quarantine failures with the full SHACL result graph. Do not load valid triples from an invalid high-consequence record unless partial acceptance is an explicit contract.

Resolution consumes staged records and emits versioned canonical membership decisions. Enrichment can then attach controlled-vocabulary mappings, geospatial normalization, or inferred classes. Generated enrichment remains derived data with its own activity and dependency links. Never write inferred triples into the same indistinguishable graph as source assertions.

Publish through immutable releases. Build a candidate graph or dataset, execute validation and competency-query tests, calculate counts and checksums, then move a release pointer atomically. Readers should see either the previous complete release or the new complete release, not half of each. Retain a manifest listing source snapshots, mapper versions, ontology and shapes releases, reasoner configuration, assertion counts, exceptions, and approval identity.

Idempotency is tested, not assumed. Replaying the same artifact and configuration should not create duplicate assertions, entities, or provenance activities unless the activity intentionally records each execution. Out-of-order events require explicit policy. A newer transaction may describe an older valid-time fact; processing order alone must not decide current truth.

Deletions are dependency operations. Removing a source artifact can retract mapped assertions, alter clusters, invalidate inferred triples, and change indexes. Compute an impact plan before publication and preserve legal holds. A tombstone may be required so a deleted upstream record is not reintroduced by a stale snapshot.

Observability should follow the stages: records received and parsed, mapping coverage, validation results by shape, candidate-pair and merge rates, inference volume, release latency, changed-query results, and orphaned provenance. Attach a correlation identifier from source artifact through release. Alert on semantic anomalies such as a sudden collapse in relationship density, not only process crashes.

The simplest first implementation can use files, a queue, deterministic functions, and a graph store. Sophisticated orchestration is secondary to immutable inputs, typed outputs, and reproducible promotion. The pipeline succeeds when a team can rebuild a release, explain every rejected item, and trace every published claim without relying on mutable operator memory.