diff options
| author | Ralph Amissah <ralph.amissah@gmail.com> | 2026-09-11 10:23:25 -0400 |
|---|---|---|
| committer | Ralph Amissah <ralph.amissah@gmail.com> | 2026-09-12 12:14:39 -0400 |
| commit | b6b2ff06054edb20c880aff26b8b02d8f093fcc2 (patch) | |
| tree | 6cc5abb9323808e28227a1791a8717ef4e00485f /src/sisudoc/outputs | |
| parent | ocda: .ssp should have all sections of abstraction (diff) | |
ocda: the abstraction carries the document metadata output reads
.ssp format v0.1 -> v1.1, and the .ocda.db with it: they are two
serialisations of one format and now move together.
The gap this closes, measured against the writers' read set: of the 59
conf_make_meta fields outputs/io_out/ reads, 29 were in neither
artefact. Five are conf.* and stay out, being site and run scoped (urls,
papersize, the search db filename): the same abstraction published to
two sites must take each site's. Of the rest, three are never assigned
anywhere (title_short, publisher, original_publisher; read only by
sqlite.d, always empty) and one is a copy of a field already carried
(title_sub = title_subtitle), so 17 properties actually had to travel
and now do:
make breaks, footer, home_button_text
meta title.edition, date.added_to_site, language.document_char,
original.{title,source,language,language_char},
rights.copyright_{text,translation,illustrations,
photographs,cover,audio,video}
make splits by when it acts, which is worth keeping in mind: these three
are read in outputs/io_out/ and must travel, while italics, bold,
emphasis, substitute and headings are read at parse time and their
effect is already in the objects.
New @source block, and source.* rows in the database, so a reader can
say which markup an abstraction came from rather than working from
stale content in silence:
language the document's own
languages the pod's list, which is what the inter-language links
in html need and neither artefact carried
digest sha256 of the .sst; equals its digests.txt entry
The database adds source.ssp_digest, the sha256 of the .ssp it was built
from, since a file cannot hold its own hash. So the chain
.sst -> .ssp -> .ocda.db is checkable end to end.
The database's metadata table is now filled from the header blocks the
.ssp gives back, not from a second list read off doc_matters. One list,
in ssp.d: a property added there arrives in the database with nothing
else changed, and one that is not in the .ssp cannot be in the database
at all. That was the last place the two could drift; the objects stopped
being able to on 2026-09-07.
Version is checked on load. The major part must match, a newer minor is
accepted (a minor bump only adds properties, and an unknown property
line is ignored). A v0.1 artefact is now refused with a message saying
to regenerate it, rather than loading half populated.
test-abstraction-db.sh now compares the two artefacts' header blocks
property by property, 36 per document on the wealth of networks, where
it previously only checked that a schema.version row existed. Verified
non-vacuous: dropping one property is reported.
Reference regenerated, and the whole diff is this change and nothing
else: 320 lines added, 35 removed over 35 files, being 35 format lines
changed, 35 @source blocks (4 lines each), 35 language.document_char, 35
home_button_text (it has a default), 18 breaks, 16 footer,
4 date.added_to_site, 1 title.edition, 1 original.source.
Existing outputs unaffected.
(assisted by Claude-Code)
Diffstat (limited to 'src/sisudoc/outputs')
| -rw-r--r-- | src/sisudoc/outputs/io_out/create_ocda_db.d | 94 |
1 files changed, 36 insertions, 58 deletions
diff --git a/src/sisudoc/outputs/io_out/create_ocda_db.d b/src/sisudoc/outputs/io_out/create_ocda_db.d index 8f915fa..f184854 100644 --- a/src/sisudoc/outputs/io_out/create_ocda_db.d +++ b/src/sisudoc/outputs/io_out/create_ocda_db.d @@ -61,13 +61,17 @@ template spineAbstractionDb() { import d2sqlite3; import sisudoc.outputs.io_out.paths_output; import sisudoc.ocda.io_in.paths_source; - /+ ↓ the abstraction is passed in rather than taken from doc, because the + import sisudoc.ocda.abstraction.ssp : ssp_format_version; + /+ ↓ the document is passed in rather than taken from doc, because the caller hands over one that has been through the .ssp: written as text and read back. so the database is a function of the .ssp and cannot - carry anything the .ssp does not + carry anything the .ssp does not, header blocks included: the metadata + table below is filled from what came back, not from a second list read + off doc_matters, which is how the two drifted apart before +/ - void spineAbstractionDb(D,A)(D doc, A doc_abstraction) { + void spineAbstractionDb(D,S)(D doc, S ssp_doc) { auto doc_matters = doc.matters; + auto doc_abstraction = ssp_doc.abstraction; /+ ↓ output path: beside the pod's zip and digests, not under <lang>/abstraction/. the database carries its own images, so it is a distributable artefact in its own right and belongs with the others: @@ -259,13 +263,18 @@ template spineAbstractionDb() { SELECT role, name, bytes, sha256, width, height FROM files ORDER BY role, name; "); - /+ ↓ populate metadata +/ + /+ ↓ populate metadata, from the header blocks the .ssp gave back. + one list, in ssp.d, not two: a property added to the .ssp header + arrives here without anything being changed, and one that is not in + the .ssp cannot be here at all. the keys are the .ssp's own + (title.main, make.doc_type, doc_has.images), prefixed by their block + where the block name is not already part of the key. + +/ db.run("BEGIN TRANSACTION"); auto meta_stmt = db.prepare( "INSERT INTO metadata (key, value) VALUES (:key, :value)" ); - auto meta = doc_matters.conf_make_meta.meta; void insertMeta(string key, string value) { if (value.length > 0) { @@ -275,62 +284,31 @@ template spineAbstractionDb() { meta_stmt.reset(); } } + void insertBlock(string prefix, string[] keys_in_order, string[string] kv) { + foreach (k; keys_in_order) { + if (k in kv) { insertMeta(prefix ~ k, kv[k]); } + } + } - /+ ↓ what this file is, so a reader can tell before it starts +/ + /+ ↓ what this file is, so a reader can tell before it starts. the + version is the .ssp's own: the two serialisations are one format and + move together + +/ insertMeta("schema.name", "sisu-abstraction-db"); - insertMeta("schema.version", "1"); - insertMeta("source.filename", doc_matters.src.filename); - insertMeta("source.language", doc_matters.src.language); - - insertMeta("title.main", meta.title_main); - insertMeta("title.subtitle", meta.title_subtitle); - insertMeta("title.full", meta.title_full); - insertMeta("title.language", meta.title_language); - insertMeta("creator.author", meta.creator_author); - insertMeta("creator.author_surname", meta.creator_author_surname); - insertMeta("creator.author_surname_fn", meta.creator_author_surname_fn); - insertMeta("creator.author_email", meta.creator_author_email); - insertMeta("creator.illustrator", meta.creator_illustrator); - insertMeta("creator.translator", meta.creator_translator); - insertMeta("date.published", meta.date_published); - insertMeta("date.created", meta.date_created); - insertMeta("date.issued", meta.date_issued); - insertMeta("date.available", meta.date_available); - insertMeta("date.modified", meta.date_modified); - insertMeta("date.valid", meta.date_valid); - insertMeta("rights.copyright", meta.rights_copyright); - insertMeta("rights.license", meta.rights_license); - insertMeta("classify.topic_register", meta.classify_topic_register); - insertMeta("classify.subject", meta.classify_subject); - insertMeta("classify.keywords", meta.classify_keywords); - insertMeta("classify.loc", meta.classify_loc); - insertMeta("classify.dewey", meta.classify_dewey); - insertMeta("identifier.isbn", meta.identifier_isbn); - insertMeta("identifier.oclc", meta.identifier_oclc); - insertMeta("language.document", meta.language_document); - insertMeta("notes.abstract", meta.notes_abstract); - insertMeta("notes.description", meta.notes_description); - insertMeta("notes.summary", meta.notes_summary); - - /+ ↓ make settings +/ - auto make = doc_matters.conf_make_meta.make; - insertMeta("make.doc_type", make.doc_type); - insertMeta("make.auto_num_top_at_level", make.auto_num_top_at_level); - insertMeta("make.auto_num_top_lv", make.auto_num_top_lv.to!string); - insertMeta("make.auto_num_depth", make.auto_num_depth.to!string); - - /+ ↓ doc_has counts +/ - insertMeta("doc_has.inline_links", doc_matters.has.inline_links.to!string); - insertMeta("doc_has.inline_notes_reg", doc_matters.has.inline_notes_reg.to!string); - insertMeta("doc_has.inline_notes_star", doc_matters.has.inline_notes_star.to!string); - insertMeta("doc_has.tables", doc_matters.has.tables.to!string); - insertMeta("doc_has.codeblocks", doc_matters.has.codeblocks.to!string); - insertMeta("doc_has.images", doc_matters.has.images.to!string); - insertMeta("doc_has.poems", doc_matters.has.poems.to!string); - insertMeta("doc_has.groups", doc_matters.has.groups.to!string); - insertMeta("doc_has.blocks", doc_matters.has.blocks.to!string); - insertMeta("doc_has.quotes", doc_matters.has.quotes.to!string); + insertMeta("schema.version", ssp_format_version); + insertMeta("source.filename", ssp_doc.source); + insertBlock("source.", ssp_doc.source_info_order, ssp_doc.source_info); + /+ ↓ the digest of the .ssp this database was built from. a file cannot + hold its own hash, so this one is not in the .ssp and is taken here: + source.digest says which markup the abstraction came from, + source.ssp_digest says which abstraction this database came from, + and the chain .sst -> .ssp -> .ocda.db is checkable end to end + +/ + insertMeta("source.ssp_digest", ssp_doc.ssp_digest); + insertBlock("", ssp_doc.meta_order, ssp_doc.meta); + insertBlock("make.", ssp_doc.make_order, ssp_doc.make); + insertBlock("doc_has.", ssp_doc.doc_has_order, ssp_doc.doc_has); meta_stmt.finalize(); /+ ↓ populate objects +/ |
