aboutsummaryrefslogtreecommitdiffhomepage
path: root/org/in_abstraction_artefacts.org
Commit message (Collapse)AuthorAgeFilesLines
* check: warn when document's languages ocn divergeRalph Amissah2026-09-221-0/+10
| | | | | | | | | | | | | | | | | | | The check runs once every language of a document has been abstracted. Providing a warning by default. --strict makes divergence a failure, taken at the end of the run so that outputs are complete and can be examined. (the name --strict is general, so that later checks can be added without a second flag). Under --parallel the profiles are appended under synchronized, and the comparison sorts a document's languages by name rather than taking them in the order the threads finished, so two runs print the same lines in the same order. The outcome is noted per language in the database as translation.ocn_aligned, for documents that have more than one language. (assisted by Claude-Code)
* ocda.db: read a document language of a databaseRalph Amissah2026-09-221-4/+67
| | | | | | | | | | | | | | | | | | | | | | | | | A database now holds every language of its document, so reading one means naming a language. Three small pieces: - spineDbLanguages says what languages a file holds. Its own template rather than part of the reader (asking a file what it holds does not instantiate the object setter and the markup regexes with it). - abstractionLoad and spineDocFromArtefact take a language and pass it to dbReadFile, which resolves it to a doc_id, takes that language document where there is one, and lists the languages and stops where there is a choice. - spineArtefactLanguages turns an artefact into the documents to build: one for a .ssp, and for a database the languages it holds filtered by --lang. The artefact loop then has a language loop inside it, so a single .ocda.db argument builds every language, as the pod it came from would. --db-round-trip takes --lang for the same reason, and ocda-db looses --parallel (would cause a race). (assisted by Claude-Code)
* ocda db: the schema gains doc_idRalph Amissah2026-09-221-8/+35
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | each file still holds one language, here groundwork for one database per document. The shape changes here and nothing merges yet: a database is still written per language, so every doc_id is 1. Real and testable on its own, where the writer and the reader together are not. documents, a row per language, is what lets one file hold a document's whole set. Everything that tells one language's rows from another's keys on documents.id. metadata is keyed on (doc_id, key) (no longer on key alone). Every language has a title and a creator, and a key-only primary key refuses the second one. schema.name and schema.version describe the file rather than a document in it, so they are written with a null doc_id and are the only rows that are. objects gain doc_id and its uniqueness widens from (section, seq) to (doc_id, section, seq). ('body', 0) exists once per language, so the narrow constraint was the thing that would have refused a second language outright. idx_objects_section leads with doc_id, or reading one language scans them all. objects.id stays a global INTEGER PRIMARY KEY, so object_images, object_links, object_anchors and object_subtoc keep their schema and their keys, and objects_fts keeps content_rowid='id'. The reader's four sweeps filter through objects rather than gaining a column of their own. outline and citable name the language and order by it first. A view over a file that may hold several languages and does not say which reads as one document and is several. The DDL is IF NOT EXISTS throughout, since a second language will open a file that already has its schema. dbReadFile takes an optional language and means "the only document in it" without one. Given none where there are several it reports the languages and stops, rather than returning the first: the round trip compares byte for byte, and a quietly wrong answer there would read as a spine fault. (assisted by Claude-Code)
* pod: read when zstd-wrappedRalph Amissah2026-09-221-1/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | sisupod using zstd compression recognised by first bytes. <doc>.sisupod: spine reads one before it writes one, so when the name changes (from .zip to .ssiupod) every pod already published stays readable. A .sisupod is one zstd frame wrapping the archive spine already builds. The reader unwraps the bytes and hands the same archive to the same parser, so every guard downstream is untouched: entry names, per-entry and total size, path depth, escape and symlinks. Which container it is comes from the first four bytes (rather than the name). A pod published as a plain .zip reads as it always did, a .sisupod reads, and either one renamed reads too. The suffix is still recognised, both spellings, for the argument and for a url. libzstd is declared rather than bound: provides the whole surface of fifteen extern C prototypes (there is nothing to generate and no upstream tree to track, which is the arrangement sqlite3 already has). dub links it with "libs": [ "zstd" ]; nix needs zstd.out rather than zstd, whose default output is the binaries and carries no library at all. A frame declares its uncompressed size in its own header, and for a pod fetched over https that number is attacker controlled. The declared size is checked against a ceiling before a buffer is asked for, a frame that will not declare one is refused, and what comes out is checked against what was promised. The ceiling is the extraction limit the archive reader already applies, so the two bounds agree. Measured: output built from a .sisupod is byte identical to output built from the same pod's .zip, 359 files over text, html, epub, odt and .ssp for three documents, live-manual's ten languages included. A truncated frame is refused and the document skipped. free_culture 2863504 -> 1099736 2.60x the_wealth_of_networks 4294022 -> 1172649 3.66x live-manual 6972418 -> 675970 10.31x (assisted by Claude-Code)
* ocda db: before writing check carried file namesRalph Amissah2026-09-221-1/+31
| | | | | | | | | | | | | | | | | | | | bugfix: content image names are now checked in a pass of their own, before anything is created or written, and one bad name refuses every image in the artefact rather than skipping the one, which is what the zip reader does with a zip. The write then re-checks containment. (prior to this an image name read out of a .ocda.db was written without checks, and such a database can be fetched over https. A name could climb with "..", and being handed to chainPath an absolute name dropped everything before it, so "/x" did not land under the image directory at all. The zip reader has guarded against this since it was written; the database reader did not). carried_names.d holds both rules: a bare filename, as an image is carried, and a relative path, for the markup and conf a database is to carry. (assisted by Claude-Code)
* epub and html: no errors left over the sample setRalph Amissah2026-09-141-2/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | epubcheck 5.3.0 over the 35 sample epubs went from 1302 remaining errors to none. The six causes for those remaining errors fixed here. most of the count was one of them: - Tables (1123 errors, 86%) where the writer emitted obsolete attributes, removed in 2014; table also now in div instead of a paragraph. also fixed: - Navigation (115). Every level 4 heading's nav entry ended "#0", the ocn of an object that has no ocn and so no id. - Duplicate ids (37). (from three unrelated sources). - The publication identifier (27 warnings). dc:identifier was a hardcoded hex string, not a UUID, and the same one in all 35 files, so two documents in one library collided. It is now UUID v5 over the document's own uid: a real RFC 4122 identifier, stable across builds and distinct per document. - Image manifest entries (15). media-type was "image/" plus the file extension, giving "image/jpg", which is not a media type; epubcheck reads it as a foreign resource and wants a fallback. There is now a mapping, which also covers svg and webp. Item ids came from the file basename, and "2bits_02_01-100.png" gave an id starting with a digit, which is not an XML name; ids are now prefixed and sanitised. - An ocn resolving to no segment (9). A poem block and its first verse share one ocn; the verses are written into a segment and the block itself is not. Also dropped from the package document: an xmlns:xsi that nothing used, and a prefix declaration for the rendition vocabulary, which is a reserved prefix and needed no declaring. The package now carries xml:lang. test-epub-validity.sh now allows no fatal and no error. Two NAV-011 warnings remain and are described in it. Three reference .ssp files change, all of them the anchor tag disambiguation above. (assisted by Claude-Code)
* some file renamesRalph Amissah2026-09-121-1/+9
|
* org renamesRalph Amissah2026-09-121-0/+1476