Imports¶
rhizodat.ingest.importer ¶
Importing a file into one store: infer, migrate, load.
The schema change an import needs (a new table, or columns the table lacks)
is a migration like any other: written as the next NNNN_import_<table>.toml
in the application's migration directory, reviewed with it, run by
Rhizodat.migrate and recorded with its time. The rows then load through
the store's Loads.load, as one unit. A failed load leaves the table as it
was; the migration stays applied, so the next import only loads.
An import concerns one store, so it needs no meta-transaction: the store's own atomicity (a transaction, an atomic partition attach, one table commit) is what makes it all-or-nothing.
ImportPlan
dataclass
¶
ImportReport
dataclass
¶
migration
instance-attribute
¶
migration: int | None
The id of the migration written and run, if the import needed one.
GraphReport
dataclass
¶
TablesGraphReport
dataclass
¶
relationships
instance-attribute
¶
relationships: list[Relationship]
Every candidate found (or given), accepted or not.
plan_import ¶
plan_import(
dat: Rhizodat,
source: str | Path,
*,
store: str,
table: str,
format: str | None = None,
null_values: Sequence[str] = DEFAULT_NULLS,
sample_rows: int | None = None,
order_by: Sequence[str] = (),
) -> ImportPlan
What importing source into store.table would do; nothing is written.
import_file ¶
import_file(
dat: Rhizodat,
source: str | Path,
*,
store: str,
table: str,
migrations: str | Path | None = None,
replace: bool = False,
format: str | None = None,
null_values: Sequence[str] = DEFAULT_NULLS,
sample_rows: int | None = None,
order_by: Sequence[str] = (),
) -> ImportReport
Import source into store.table: the migration it needs (written to the
migrations directory and run), then the rows, as one unit; replace swaps
the table's contents.
import_graph ¶
import_graph(
dat: Rhizodat,
source: str | Path,
*,
store: str,
format: str | None = None,
batch: int = 500,
) -> GraphReport
Import an RDF file (Turtle, N-Triples, N3, JSON-LD, RDF/XML, OWL) into a graph store, as one unit: resources become nodes keyed by their IRI, links edges.
A graph has no schema to migrate: labels and edge types come with the data, and a second import of the same file updates what changed instead of duplicating.
import_tables_graph ¶
import_tables_graph(
dat: Rhizodat,
sources: Mapping[str, str | Path],
*,
store: str,
keys: Mapping[str, str | None] | None = None,
min_coverage: float = 0.95,
relationships: Sequence[Relationship] | None = None,
dry_run: bool = False,
batch: int = 500,
) -> TablesGraphReport
Several tables (by name) as one graph in store: rows as nodes, references
between them as edges, inferred unless relationships are given; one unit.
rhizodat.ingest.sources ¶
Reading import sources: CSV, TSV, JSON Lines and Parquet, local or in object storage.
A source is a path or a URI pyarrow can open (s3://bucket/key, gs://,
file://). S3 credentials come from the usual AWS_* environment variables
or the URI itself (s3://ACCESS:SECRET@bucket/key?endpoint_override=host:port&
scheme=http®ion=...); the secret is masked wherever the source is shown.
Compressed CSV, TSV and JSON Lines (.gz, .bz2, .zst, ...) are read
as they are.
Inference reads every row (or the first sample_rows), with each value
seen as text: a type holds for all of them, not only for the first block.
The second pass reads the file again with those types.
infer_schema ¶
infer_schema(
source: str | Path,
*,
format: str | None = None,
null_values: Sequence[str] = DEFAULT_NULLS,
sample_rows: int | None = None,
) -> TableSchema
The columns of source with the narrowest type each holds for every row read.
read_batches ¶
read_batches(
source: str | Path, schema: TableSchema
) -> Iterator[Any]
source as pyarrow record batches with schema's types.
source_format ¶
source_format(source: str | Path) -> str
The format named by the file's suffix (a compression suffix is looked past).
rhizodat.ingest.schema ¶
The schema of an imported file: columns with a portable type.
A column's type is one of a few logical types every store can hold
(boolean, int64, float64, date, timestamp,
timestamptz, string); each store adapter maps them to its own. Types
widen along two chains, int64 → float64 and date → timestamp. Any
other mix (a flag and a number, a time with and one without a zone) is
string: text keeps every value as it was.
Column
dataclass
¶
low_cardinality
class-attribute
instance-attribute
¶
low_cardinality: bool = False
Few distinct values for many rows: a dictionary encoding pays (ClickHouse
LowCardinality); a hint, stores may ignore it.
TableSchema
dataclass
¶
join ¶
join(a: LogicalType, b: LogicalType) -> LogicalType
The narrowest type holding values of both a and b.
rhizodat.ingest.graph ¶
Reading RDF (Turtle, N-Triples, N3, JSON-LD, RDF/XML, OWL) as nodes and edges.
Each resource is one node keyed by its IRI (a blank node by _:<id>, unique
within one file). Every rdf:type of a resource is a label (the class's
local name). A literal value is a property named after its predicate's local
name; several values of one predicate (a name in two languages) are a list. A
link to another resource is an edge typed after its predicate, and the target
is a node even when the file says nothing else about it.
Names are made identifiers (lives-in → lives_in). A local name that two
vocabularies of the file share (a:name, b:name) keeps them apart with
their prefixes (a_name, b_name). Values keep their type where the graph
can hold it (integers, floats, booleans, text); dates, times and decimals are
stored as their exact ISO or decimal text.
rhizodat.ingest.relate ¶
Relationships inferred from tables, and the tables as a graph.
Several exported tables (CSV, TSV, JSON Lines, Parquet) become one graph: each
row a node labelled after its table (order_lines → OrderLine), each
reference between rows an edge.
- A table's key is a column with no missing and no repeated value:
id, else<table>_id, else the first such column (or the one given). - A column refers to a table when its name says so: its stem (the name
without
_id/_ref) is that table's name in the singular (customer_id→customers). A column ending in_idor_refthat names no table refers to its own table (manager_idinemployees). Values alone are not enough: quantities 1..5 match ids 1..5 completely. - A reference is accepted when at least
min_coverageof its values are found among the target's keys; every candidate is reported with how many matched, so a near miss (orphans in an export) is visible.
Tables are read into memory: this is for exports of moderate size.
Relationship
dataclass
¶
singular ¶
singular(name: str) -> str
customers → customer, categories → category (plain English plurals).
read_tables ¶
read_tables(
sources: Mapping[str, str | Path],
*,
keys: Mapping[str, str | None] | None = None,
) -> dict[str, Table]
Each source read with its inferred schema; keys names a table's key column
(None: the table has none) instead of finding it.
infer_relationships ¶
infer_relationships(
tables: Mapping[str, Table],
*,
min_coverage: float = 0.95,
) -> list[Relationship]
Every column that refers to a table by its name, with how many of its values
are that table's keys; accepted at min_coverage or above.
measure ¶
measure(
tables: Mapping[str, Table],
table: str,
column: str,
target: str,
) -> Relationship
How many values of table.column are keys of target: a relationship someone
proposed (a person, a model), checked against the data. Accepted when any matched.
tables_to_graph ¶
tables_to_graph(
tables: Mapping[str, Table],
relationships: Sequence[Relationship],
) -> GraphData
Rows as nodes (<table>:<key>, or <table>:<row number> without a key) and
each accepted relationship's matching values as edges.