aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/sisudoc/outputs
Commit message (Collapse)AuthorAgeFilesLines
* org files mostly headersRalph Amissah2026-10-2429-580/+435
|
* metadata page: reporting digests only provide linkRalph Amissah2026-10-011-19/+8
| | | | | | | | | | | | | | fix issue with reporting of multi-lingual document digests Set as with the .ocda.db digest taken off this page on 2026-09-18, and the same remedy: the page links <doc>.digests.txt, where each digest is taken once, of the finished file, and quotes none itself. single-language documents lose a line that was right, but for multi-language documents the last language in the run was missing here, and this cannot go wrong again. (assisted by Claude-Code)
* title is h1 class "heading title", sized above h1Ralph Amissah2026-10-012-32/+44
| | | | | | | | | html & epub (share css): with <h0> gone the title became <h1>. The title now carries class "heading title", and each stylesheet's h0 rules are h1.title rules with the same values; h0 leaves the grouped selectors, being emitted by nothing. In html the title looks as it did. (assisted by Claude-Code)
* html & epub header fixes, miscellanyRalph Amissah2026-10-011-132/+85
| | | | | | | | - adjust metadata, cleanup fixes - no <h0> realign title on h1 along with bespoke css - show group bullets (assisted by Claude-Code)
* abstraction 2.1: a poem states its object rangeRalph Amissah2026-09-301-2/+6
| | | | | | | | | | | | | | | - A poem is a container of verse, and stores its range of verse ocn, - its verse are the citable units with ocn. - A note in the last verse of a poem previously was not gathered into the endnotes section, this now is fixed Format 2.0 -> 2.1: the property is an addition, the poem is not a citable object (but contans a range of objects), its verse are (individual citable objects), and every reader checks the major version. A 2.0 database reads as having no ranges. (assisted by Claude-Code)
* search database: show all included notesRalph Amissah2026-09-301-3/+9
| | | | | | | | | | | | | | | | search database: show notes other than regular numbered (foot)notes. make sure all types of notes for all object types set the objects has.inline_notes_reg flag required for them to be rendered search results' html (doc_objects.body) show notes only where an object's has.inline_notes_reg flag is set, which occurred for numbered notes only - only numbered notes, regular footnotes were included - and notes in some objects were omitted (quotes and verse) (assisted by Claude-Code)
* notes: editor's notes, ~[* ]~ & ~[+ ]~, as seriesRalph Amissah2026-09-304-7/+43
| | | | | | | | | | The markup documents ~[* note ]~ and ~[+ note ]~ as editor's notes, each a separately numbered series, and a bare ~[ note ]~ which sisu put in the asterisk series. They are included as notes. Each series is numbered through the document, *1, *2 ... and +1, +2 ..., apart from the author's notes. (assisted by Claude-Code)
* markdown output, --markdown (--md)Ralph Amissah2026-09-285-1/+673
| | | | | | | | | | | | | | | | | | | | | A plain-text output that keeps the object numbers, written as CommonMark with GFM pipe tables, one .md per document-language in <lang>/markdown/, (linked from the metadata page) with images into that shared directory itself, skipping any already there, so --markdown alone still gives a complete tree. provided as an alternative to --text rather than a replacement --markdown produces a document that renders, (both provide objext numbering survives both. Every object carries an anchor and a superscript number linking to itself, so a citation by object number is a link in any markdown renderer. --text may still be of value to read in a terminal A note is written where it is referenced, with a link back to the object it came from, rather than gathered into an endnotes section. (assisted by Claude-Code)
* typst output --typist & --pdf that makes the pdfRalph Amissah2026-09-284-1/+918
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | typst provides a new pdf build path replacing latex as the primary pdf builder. --typst (--typ) writes the .typ; --pdf writes the .typ and compiles it by spawning `typst compile`. (--pdf is redefined, being typst to pdf rather than latex). --latex is unchanged and writes the .tex; One .typ per document-language (not one per paper size). The paper and the orientation are read from sys.inputs with a default, so the same file compiles to every paper --set-papersize asks for, the pdfs keep their original names. Spine's paper names (derived from what suited latex) and typst's are different vocabularies and are mapped; a paper typst does not have is named and skipped. With the flag --pdf spine invokes a typesetter to generate the pdf output directly, which is new (the latex path writes a .tex and leaves xelatex to the caller). If typst is not available on the machine it is reported absent, naming the binary and printing the command, and the run continues, the .typ is written. Several useful features of an object-centric pdf are easily met, usually being a few lines each. - The object number goes in the margin as one `place` at a coordinate inside the object's own block, carrying the label every citation points at. - A note is a real footnote at the point of reference, with the document's own number and one link back to the object. - The table of contents locates by object number rather than by page, so it holds true across every paper and every setting. That typst compiles silently across the sample set means every internal link in every document resolves (--pdf over the collection with two paper sizes gives 144 pdfs, none skipped and none failed, each the page size its name claims and each tagged). paths: .typ, pdf named in same way as .tex based built pdfs so site linkage works either way flake.nix: added a typst dev shell, for .typ pdf typesetter - dsh-typst-pdf providing a pdf typsetter for spine's .typ. - dsh-latex-pdf provides xelatex for .tex which can still be written. (assisted by Claude-Code)
* uid and paths separator "~" in place of ":"Ralph Amissah2026-09-232-11/+11
| | | | | | | | | | | | | | | | | | | need a character to split filenames on in certain circumstances. there problems with use of a colon in filenames, which is legal on posix but not on Windows."~" fits the bill better being legal on every filesystem of interest and is unreserved in rfc 3986, needing no escaping in a url. The reference abstraction is renamed, its content unchanged. Over the sample collection two filenames move and nothing else does: the abstraction and database digests are identical, the archive's members and their sizes are unchanged, and only the member order shifts, "~" collating after letters where ":" sorted before them. The document's epub dc:identifier is a v5 uuid derived from the uid, so it changes for those filenames. Every document's uid changes in the search database (spine.search.db). (assisted by Claude-Code)
* cleanup: misc. (previously deferred)Ralph Amissah2026-09-223-3/+22
| | | | | | | | | | | | | | | | | | | | | dr_document_make in three identifier spellings becomes document_make, which is what the file has been called since the rename. The mixin import audit, a template declares its imports at template scope and they are then visible in every scope it is mixed into (which is how a template-level split once hijacked a UFCS lookup in code that had not been touched). Disabling htmlSnippet's imports and rebuilding names the consumers that were relying on them rather than on their own: across twelve mixin sites, exactly one. metadata.d now imports the `to` it uses. The templates keep their imports, their own functions needing them, and narrowing those is a change of its own rather than a cleanup. And the standing FIX in source_pod.d: the insert digest line for a non pod source recorded the insert's path, where every other line in digests.txt is a filename and a digest naming a file that is not in the pod under that name cannot be checked against it. (assisted by Claude-Code)
* ocda db: carry catalogues & text blobs compressedRalph Amissah2026-09-221-17/+88
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The pod carries tools/po4a and the database (until now) did not, so a pod written back out of a database came back a lossy copy, without its translation catalogues. the database now carries them, walked and name-checked exactly as the pod writer walks and checks them, once on the last language. They are compressed to save space, the test sample carrying 6.25 MB of catalogue as read would have made that one document's database two thirds larger. Files gains a compression column: NULL or 'none' for the bytes as read, 'zstd' for a frame. What to compress is decided by role, not by size (source, conf, manifest and tools are text and are compressed; images being compressed already are not). The same kind of file is always stored the same way. Document objects are untouched and stay raw: objects_fts is external content over objects.text and reads that column directly. bytes and sha256 remain those of the original file, so every digest check, digests.txt line and source.digest rebuilt from these rows works unchanged, and a reader that does not decompress can still say what it is looking at. dbReadFiles decompresses, so no caller learns how a blob is stored. It asks pragma_table_info whether the column is there at all: a database written before it reads as raw. A row that claims zstd and will not decompress is named and left out, rather than handed back as a frame where markup should be. live-manual: 9,490,432 bytes without the catalogues, 9,908,224 with them. Output from the materialised pod is identical to output from the original pod, 679 files each side, no differences. (assisted by Claude-Code)
* ocda db: now carry markup source, conf & manifestRalph Amissah2026-09-221-0/+82
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The database already carried the images, because without them no output can be produced. It now carries the markup as well, and with it the document's configuration and the manifest as the author wrote it. That is the difference between a serialised abstraction and a document source. An abstraction can be rendered but not re-parsed, and a reader who wants to correct a sentence needs the sentence as written. With these a pod can be written back out of the sqlite-file. Three new values in the existing role column, no schema change: the format stays 2.0. Source rows are named by their path within the pod, media/text/<lang>/<file>, and not by bare filename as images are. Every language of a document has a file of the same name, so bare names would collide under UNIQUE(role, name) and nine of ten would be dropped without a word. The path is also what a pod materialised from this database has to be told. Bytes stored as read with the digest over them, so that each row is checkable against the line source.digest was built from. Also build.spine_version, a file level row saying which spine wrote the file: not a property of the document, and the one thing a file cannot be asked for afterwards. The reader skips the build. prefix as it skips schema. and translation., or it would come back inside the document header and the round trip would differ. (assisted by Claude-Code)
* check: warn when document's languages ocn divergeRalph Amissah2026-09-221-1/+51
| | | | | | | | | | | | | | | | | | | The check runs once every language of a document has been abstracted. Providing a warning by default. --strict makes divergence a failure, taken at the end of the run so that outputs are complete and can be examined. (the name --strict is general, so that later checks can be added without a second flag). Under --parallel the profiles are appended under synchronized, and the comparison sorts a document's languages by name rather than taking them in the order the threads finished, so two runs print the same lines in the same order. The outcome is noted per language in the database as translation.ocn_aligned, for documents that have more than one language. (assisted by Claude-Code)
* digests: db digest taken once on file completionRalph Amissah2026-09-222-34/+28
| | | | | | | | | | One file for the document (including all languages) with digest of the document. digests.txt lists it once, beside the shared images, taken when the last language has been written. digests.txt carries all digests. (assisted by Claude-Code)
* ocda db: one <doc>.ocda.db holding every languageRalph Amissah2026-09-221-10/+29
| | | | | | | | | | | | | | | | | | | The file is named for the document, not for one of its languages (and is written a language at a time: ten calls fill one file for a ten language document). The removal on entry was right when a call wrote the whole file. It now happens on the first language of the document, taken from the manifest rather than from whatever order the loop ran in, so nine languages are no longer written and thrown away. Also a partial unique index for the file level metadata rows. In SQL no null equals any other null, so UNIQUE(doc_id, key) does not constrain the rows written with a null doc_id, and schema.name and schema.version were inserted once per language: ten rows each, with nothing for INSERT OR REPLACE to replace. (assisted by Claude-Code)
* ocda db: the schema gains doc_idRalph Amissah2026-09-221-32/+111
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | each file still holds one language, here groundwork for one database per document. The shape changes here and nothing merges yet: a database is still written per language, so every doc_id is 1. Real and testable on its own, where the writer and the reader together are not. documents, a row per language, is what lets one file hold a document's whole set. Everything that tells one language's rows from another's keys on documents.id. metadata is keyed on (doc_id, key) (no longer on key alone). Every language has a title and a creator, and a key-only primary key refuses the second one. schema.name and schema.version describe the file rather than a document in it, so they are written with a null doc_id and are the only rows that are. objects gain doc_id and its uniqueness widens from (section, seq) to (doc_id, section, seq). ('body', 0) exists once per language, so the narrow constraint was the thing that would have refused a second language outright. idx_objects_section leads with doc_id, or reading one language scans them all. objects.id stays a global INTEGER PRIMARY KEY, so object_images, object_links, object_anchors and object_subtoc keep their schema and their keys, and objects_fts keeps content_rowid='id'. The reader's four sweeps filter through objects rather than gaining a column of their own. outline and citable name the language and order by it first. A view over a file that may hold several languages and does not say which reads as one document and is several. The DDL is IF NOT EXISTS throughout, since a second language will open a file that already has its schema. dbReadFile takes an optional language and means "the only document in it" without one. Given none where there are several it reports the languages and stops, rather than returning the first: the round trip compares byte for byte, and a quietly wrong answer there would read as a spine fault. (assisted by Claude-Code)
* pod: generate the po4a configuration, --po4a-cfgRalph Amissah2026-09-222-0/+188
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A document's po4a configuration is derived rather than kept. Spine already knows both halves without looking: the manifest says which languages, and the markup says which files, through the << lines the parser follows. Derived, it cannot disagree with the document, where a hand-kept file eventually does: a language added to the manifest and forgotten in the Makefile is a translation nobody updates. It prints, rather than writing. Spine does not write into a source tree, and the file belongs beside the catalogues in the pod it describes: spine --po4a-cfg <pod> > <pod>/tools/po4a/po4a.cfg Three things the Makefile it replaces knew, which had to be found by running po4a rather than by reading: --keep 0 is the difference between regenerating a document and destroying it. po4a will not write a translated file below a completeness threshold, and that threshold defaults to 80%. A partly translated document is the normal state of one being worked on, and sisu covers it: an untranslated string falls back to the source text. Measured at the default, 181 of live-manual's 189 translated files would be discarded, and nine languages would collapse to eight surviving files, reading as the document reverting to its source language rather than as a failure. index.html.in is not markup and no << line names it, so a configuration built from the manifest and the insert list alone drops it, and with it index.html.in.pot and its nine .po files at the next regeneration. It is found by asking whether the source language has one. neverwrap is not set, and the measurement is recorded beside the code: setting it turns 889 of 1126 strings fuzzy in four languages, and po4a does not use a fuzzy translation, so Catalan's coverage would fall from about 46% to about 12%. Tidy formatting is not worth that. The run-complete banner is suppressed for this action. The configuration goes to stdout and the banner would otherwise be part of the file, which is what the round trips already avoid by leaving early; this one prints from inside output processing and cannot. (assisted by Claude-Code)
* pod: carry tools/po4a, the translation working setRalph Amissah2026-09-221-0/+41
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A pod now carries its catalogues, so a translation can be updated rather than only retyped: providing the .pot and .po (the means of maintaining it). A directory walk, where every other thing the pod writer carries is named. A catalogue set is whatever the translator has, and enumerating .pot and .po files would mean the writer deriving the document's languages a second time, from a second place, to say what it already reads from the manifest. Done once, on the last language, as the markup blocks are. The tree is not language specific, so a pass for each language would re-read all of it, 5.5 MB ten times over for live-manual, into an archive the next pass overwrites. Every name is checked before it is carried, by the rule the pod reader applies to an archive it is handed. The writer is reading a directory someone else may have filled, which is the reason the reader checks. The cost is small, and it is the whole argument for compressing the archive as a stream rather than per member: uncompressed content 6972418 -> 13167489 +89% the .sisupod 692942 -> 891388 +28.6% 5.5 MB of catalogues cost 198 KB, because the .po files largely repeat markup already in the archive and whole-stream compression sees it. Per-member deflate could not. (assisted by Claude-Code)
* pod: set entry limit for the writer and readerRalph Amissah2026-09-221-0/+38
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Both now take the number from one constant now set at 10,000, the writer checks it before the archive is written. Previously the number set for the pod reader was less than 500 members. The writer had no limit, (so spine could write a pod it would then refuse to read, reporting too many entries: a good file that looks corrupt, and only for a document with enough parts). 500 was set when a pod held markup and images. A pod carrying translation catalogues has a different arithmetic, and the count follows from how many files a document is made of times how many languages it has, not from how large the document is: the_wealth_of_networks 213,405 words, 1 file per language 12 languages -> 49 entries live-manual 24,724 words, 20 files per language 12 languages -> 506 entries The big book is not what runs into this; the modular manual is. live-manual stood two languages from an unreadable artefact. 10,000 covers a hundred-insert manual in thirty languages, about 6,200 entries, with room. It is deliberately well clear of any real document rather than snug above the largest one known: the two errors are not comparable, since too low refuses a legitimate document with a message that reads as corruption, while too high defers to a size cap a moment later. The count is the weakest of the three guards and is not what bounds resource use; the per entry and total size caps do that, both before a byte is written. Refusing also removes any archive an earlier language left. The writer runs once per language and only the last pass holds every language, so it is the last that goes over, and the passes before it wrote smaller archives that passed. Without this, refusal left a pod missing a language: an artefact that reads perfectly well and is wrong. A missing file is an error someone notices. The arithmetic is recorded beside the constant so the next person can re-derive the number rather than guess at it. (assisted by Claude-Code)
* pod: write <doc>.sisupod (single zstd frame)Ralph Amissah2026-09-222-8/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | The pod archive spine writes is now named .sisupod and carries the same archive inside one zstd frame. The reader has accepted both since the previous commit, so every pod already published stays readable and this changes only what is written. free_culture 2863504 -> 1100180 live-manual 6972418 -> 692942 10.06x The wrap is at the pod call site and not in createZipFile, which also writes every epub and every odt. Best compression from whole-stream (rather than per-member compression). (for live-manual per-member deflate manages about 3x compared to 10x for whole pod content compression). The suffix is named as pod_archive_suffix. --pod-compression sets the level, 19 by default: a pod is written once and fetched many times (692942 bytes at 19, 837703 at 9, 2054365 at 1). (A value that is not a number warns and the default is used). Output built from a written .sisupod is byte identical to output built from the markup it came from. (assisted by Claude-Code)
* pod: read when zstd-wrappedRalph Amissah2026-09-221-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | sisupod using zstd compression recognised by first bytes. <doc>.sisupod: spine reads one before it writes one, so when the name changes (from .zip to .ssiupod) every pod already published stays readable. A .sisupod is one zstd frame wrapping the archive spine already builds. The reader unwraps the bytes and hands the same archive to the same parser, so every guard downstream is untouched: entry names, per-entry and total size, path depth, escape and symlinks. Which container it is comes from the first four bytes (rather than the name). A pod published as a plain .zip reads as it always did, a .sisupod reads, and either one renamed reads too. The suffix is still recognised, both spellings, for the argument and for a url. libzstd is declared rather than bound: provides the whole surface of fifteen extern C prototypes (there is nothing to generate and no upstream tree to track, which is the arrangement sqlite3 already has). dub links it with "libs": [ "zstd" ]; nix needs zstd.out rather than zstd, whose default output is the binaries and carries no library at all. A frame declares its uncompressed size in its own header, and for a pod fetched over https that number is attacker controlled. The declared size is checked against a ceiling before a buffer is asked for, a frame that will not declare one is refused, and what comes out is checked against what was promised. The ceiling is the extraction limit the archive reader already applies, so the two bounds agree. Measured: output built from a .sisupod is byte identical to output built from the same pod's .zip, 359 files over text, html, epub, odt and .ssp for three documents, live-manual's ten languages included. A truncated frame is refused and the document skipped. free_culture 2863504 -> 1099736 2.60x the_wealth_of_networks 4294022 -> 1172649 3.66x live-manual 6972418 -> 675970 10.31x (assisted by Claude-Code)
* odt: office:meta from the document, not the runRalph Amissah2026-09-222-72/+16
| | | | | | | | | | | | | | | | | | | | | | | | | Three faults in four lines of the odt metadata. meta:creation-date and dc:date were taken from generated_time, the run's own clock, so two runs over unchanged markup produced different odt bytes. They now carry the document's own instant, so an odt is a function of its source. That was the last clock stamp in any writer: every output format is now reproducible. generated_time is a human-readable run stamp, of the form "2026-9-17 [38/3] 18:39:35", with unpadded numbers and an iso week in brackets. It is not a valid xsd:dateTime, which is what ODF asks for in both of those fields, so what was written there was malformed as well as unstable. dc:language now carries the document's own language tag. The instant is doc_matters.modified_utc, moved there from epub3.d in this commit: epub3 needs the same value for its dcterms:modified, and one definition cannot drift from the other. generated_time now has no caller and is left in place. (assisted by Claude-Code)
* metadata: use document's languageRalph Amissah2026-09-221-2/+6
| | | | | | | Every metadata page body element carried lang="en" xml:lang="en" as a literal. It now carries the document's own language tag. (assisted by Claude-Code)
* epub: dcterms:modified timestamp, reproducibilityRalph Amissah2026-09-221-3/+62
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | timestamp dcterms:modified from the document, not the clock EPUB3 requires exactly one dcterms:modified on the package It is now read in order: SOURCE_DATE_EPOCH where the environment sets it, which is the reproducible-builds convention and lets a build pin every artefact of one run; then the document's own date.modified; then date.published. The clock remains the last resort, for a document that says nothing about when it is from. A document date is loose markup. "1991", "2006-03" and "2021-12-00" all occur in the sample set, and a month or day of 00 is not a date. A missing or zero part becomes 01, and a day out of range for its month falls back to the first, so what is written is always a valid CCYY-MM-DDThh:mm:ssZ. Over the sample set two consecutive runs now produce all 36 epubs byte identical, and epubcheck reports no change: 0 fatal, 0 errors. (previously taken from the clock at build time. Two runs over unchanged markup therefore produced different epub bytes, so no comparison of epub output could tell a real change from the seconds between two builds. It also meant an epub was a function of when it was built rather than of what it was built from.) (assisted by Claude-Code)
* conf: read conf/document_make, drop the dr_ prefixRalph Amissah2026-09-221-4/+4
| | | | | | | Spine now looks for conf/document_make, and bundles that name. (the dr_ prefix was a sisu-era name to distinguish it) (assisted by Claude-Code)
* xml: markup language attributes take a bcp 47 tagRalph Amissah2026-09-222-10/+10
| | | | | | | | | | | | | | | | Spine names a language as the directory under media/text/ is named, which for a region is the posix form with an underscore, pt_BR. A lang, xml:lang or dc:language value is a BCP 47 (RFC 3066) tag and wants a hyphen, pt-BR. Spine put its own code there unchanged, so a document in a region language produced an invalid attribute in every one of its xhtml files: 138 epubcheck errors for a document of ten segments and a navigation file. src.language_tag is the same language as a tag. The attributes and the dc metadata now read it. Output paths keep src.language, since a path is named after the directory, not after the tag. (assisted by Claude-Code)
* pod digests: multilingual document digests sortedRalph Amissah2026-09-221-7/+16
| | | | | | | | | | | | | | Fixes for three faults in the insert (.ssi) digests. The key was the language captured from whichever path resolved the insert. It did not change across the loop over languages, so every language's inserts were filed under one language. The key is now the language being digested, as the digest of the primary file above it already did. - digest is taken after the check that the file exists. - digest is now taken inside the guard. - non-pod path now correctly keyed on the document's own language. (assisted by Claude-Code)
* images alt text, and epub says what it isRalph Amissah2026-09-144-29/+286
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Four things, not visible in an error count: - Images had no alt text, even where markup carried the text, and the field alt text is present, incorrectly used in code; fixed as the alt. { sm_tux.png 64x80 "Gnu/Linux - a better way" }image an image with nothing to say for itself gets alt="" - Images with no dimensions were incorrectly written width="0" height="0", fixed - Accessibility metadata, which EPUB Accessibility 1.1 requires: schema:accessMode, accessibilityFeature and accessibilityHazard, plus accessModeSufficient and accessibilitySummary, which epubcheck does not test for; used by - readers looking for a book it can use, - anyone distributing into the EU. Every value is derived from the document, not asserted: a document with no images says so, one whose images all carry alt text claims alternativeText and textual sufficiency, one where an image says nothing claims neither. Nothing claims WCAG conformance, which would be a claim about an evaluation that has not happened. - DPUB-ARIA roles beside the structure spine already knew about. doc-toc, doc-endnotes, doc-bibliography, doc-glossary and doc-index on the sections, doc-noteref on a note reference and doc-footnote on the note it points at. "doc_endnotes" as a class name means something to a stylesheet and nothing to a reading system. Not doc-biblioentry: DPUB- ARIA 1.1 deprecated it. These are plain ARIA, so the html gets them too. Also dealt with here: - Spine makes certain sections such as endnotes, bookindex, ... and a heading that has a name for itself cannot share the same name, if it does the name is now disambiguated, disambiguated. (disambiguation similar to that used for a repeated anchor tag) - Endnote anchors fixed to use "id" (being one unique tag in the document that is target of every reference to it)instead of obsolete form (they were <a name="note_1">. "name" on an <a>). Book index markers keep "name": the same marker sits on every object an entry names, 296 times in one document, and those are markers of where a term occurs rather than distinct targets. (assisted by Claude-Code)
* epub and html: no errors left over the sample setRalph Amissah2026-09-144-69/+279
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | epubcheck 5.3.0 over the 35 sample epubs went from 1302 remaining errors to none. The six causes for those remaining errors fixed here. most of the count was one of them: - Tables (1123 errors, 86%) where the writer emitted obsolete attributes, removed in 2014; table also now in div instead of a paragraph. also fixed: - Navigation (115). Every level 4 heading's nav entry ended "#0", the ocn of an object that has no ocn and so no id. - Duplicate ids (37). (from three unrelated sources). - The publication identifier (27 warnings). dc:identifier was a hardcoded hex string, not a UUID, and the same one in all 35 files, so two documents in one library collided. It is now UUID v5 over the document's own uid: a real RFC 4122 identifier, stable across builds and distinct per document. - Image manifest entries (15). media-type was "image/" plus the file extension, giving "image/jpg", which is not a media type; epubcheck reads it as a foreign resource and wants a fallback. There is now a mapping, which also covers svg and webp. Item ids came from the file basename, and "2bits_02_01-100.png" gave an id starting with a digit, which is not an XML name; ids are now prefixed and sanitised. - An ocn resolving to no segment (9). A poem block and its first verse share one ocn; the verses are written into a segment and the block itself is not. Also dropped from the package document: an xmlns:xsi that nothing used, and a prefix declaration for the rendition vocabulary, which is a reserved prefix and needed no declaring. The package now carries xml:lang. test-epub-validity.sh now allows no fatal and no error. Two NAV-011 warnings remain and are described in it. Three reference .ssp files change, all of them the anchor tag disambiguation above. (assisted by Claude-Code)
* epub, html: a named anchor, and subtitle fixesRalph Amissah2026-09-142-16/+15
| | | | | | | | fixes related to: - named anchor id - subtitles, emit only where they exist (assisted by Claude-Code)
* epub, html: correctly identify code block as blockRalph Amissah2026-09-142-10/+11
| | | | | | | | - wrapper set to <div>, with class unchanged, and the six stylesheets select div.code where they selected p.code. - invented element <codeline> removed (assisted by Claude-Code)
* epub, html: reference issues fixesRalph Amissah2026-09-142-18/+10
| | | | | | | | - _part_eof.xhtml was declared in the manifest and never written. - image_sys/bullet_09.png was referenced as a background image, a throwback to the Ruby version of sisu, never used here. (assisted by Claude-Code)
* epub, html: document heads and ids, fixesRalph Amissah2026-09-142-6/+13
| | | | | | | One document head per file, and ids must be unique. No remaining fatal errors in the epub in the marup sample set. (assisted by Claude-Code)
* epub, html, db: use of elements & attributes, fixRalph Amissah2026-09-143-120/+121
| | | | | | | fixed, four things used that XHTML does not have. (search db sisudoc-spine-search-cgi results share styling) (assisted by Claude-Code)
* epub: mimetype entry comes first in the archiveRalph Amissah2026-09-141-3/+18
| | | | | | | | | | | | | | | OCF requires the mimetype entry to be the first in the zip and stored uncompressed, so that a reader can tell what the file is by looking at a known offset without parsing the archive. The index is now set to: 0 for the mimetype, 1 upward for everything else as it is added. mimetype Stored first META-INF/container.xml Stored OEBPS/toc_nav.xhtml Stored ... (assisted by Claude-Code)
* epub: manifest lists files, spine orders filesRalph Amissah2026-09-141-57/+46
| | | | | | | | | | several fixes - for epub manifest is the list of files in the container, the spine is the reading order of those files, fixed. - The guide is dropped. It is EPUB2, optional in EPUB3 - numerous small fixes (assisted by Claude-Code)
* epub, html: metadata text clear of markupRalph Amissah2026-09-132-30/+53
| | | | | | | | special_characters_text escapes the markup characters and then puts <br /> back in for a line break. That is right for element content and wrong in metadata. fix (assisted by Claude-Code)
* epub, html, odt: namespace names as identifiersRalph Amissah2026-09-133-17/+17
| | | | | | | | | | | | | | namespace names are identifiers, not addresses, fixes A namespace URI is an opaque identifier. www.w3.org xhtml, xlink, MathML, rdf-syntax-ns, xml-events, XMLSchema, XMLSchema-instance, xforms, css3-text, data-view, and the xhtml1 strict DTD system identifier purl.org dc/elements/1.1, dc/terms openoffice.org office, writer, calc, table, report docs.oasis-open the ODF meta pkg and odf class identifiers (assisted by Claude-Code)
* some file renamesRalph Amissah2026-09-129-16/+16
|
* ocda: heading cross reference linkingRalph Amissah2026-09-121-2/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The segment a cross reference to a heading lands on, gixed upstream in ocda, where the field is created, rather than inferred by the loader. A heading above level 4 opens no html segment of its own, so a link to it has to land on the level 4 heading that follows. The build loop cannot know that when it reads the heading, and says so in a comment of its own: "for html segname need following lv4 not yet known". It back-fills tag_assoc when the level 4 heading arrives (lv0to3_tags), and the answer then lives only in that map. tags.segment_lv4_is now holds it, resolved in a pass over the finished head and body sections: walking backwards, the last level 4 heading seen is the next one for everything above it, which is the same answer the back-fill gives. Emitted in the .ssp only where it differs from .segment_html_is, which is every heading at level 4 or below, so it is sparse: 317 lines over the 35 document reference set, none removed. Carried in the database as a column of its own, and read back by both readers. docHasFromAbstraction reads it instead of working it out. With that, and on top of the two defect fixes, a document loaded from an artefact and one parsed from markup agree: key sets identical, all 35 documents values 3 entries differ of some 30,000, all of them _the_title, where the abstraction supplies an epub segment the parser leaves unset link targets all 1,651 agree, against 89 differing before any of this work and 1 after the defect fixes alone Format stays v1.1. That version is new in this same run of work and nothing outside spine has read it, so this belongs in it rather than in a bump of its own. (assisted by Claude-Code)
* ocda: the abstraction carries the document metadata output readsRalph Amissah2026-09-121-58/+36
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | .ssp format v0.1 -> v1.1, and the .ocda.db with it: they are two serialisations of one format and now move together. The gap this closes, measured against the writers' read set: of the 59 conf_make_meta fields outputs/io_out/ reads, 29 were in neither artefact. Five are conf.* and stay out, being site and run scoped (urls, papersize, the search db filename): the same abstraction published to two sites must take each site's. Of the rest, three are never assigned anywhere (title_short, publisher, original_publisher; read only by sqlite.d, always empty) and one is a copy of a field already carried (title_sub = title_subtitle), so 17 properties actually had to travel and now do: make breaks, footer, home_button_text meta title.edition, date.added_to_site, language.document_char, original.{title,source,language,language_char}, rights.copyright_{text,translation,illustrations, photographs,cover,audio,video} make splits by when it acts, which is worth keeping in mind: these three are read in outputs/io_out/ and must travel, while italics, bold, emphasis, substitute and headings are read at parse time and their effect is already in the objects. New @source block, and source.* rows in the database, so a reader can say which markup an abstraction came from rather than working from stale content in silence: language the document's own languages the pod's list, which is what the inter-language links in html need and neither artefact carried digest sha256 of the .sst; equals its digests.txt entry The database adds source.ssp_digest, the sha256 of the .ssp it was built from, since a file cannot hold its own hash. So the chain .sst -> .ssp -> .ocda.db is checkable end to end. The database's metadata table is now filled from the header blocks the .ssp gives back, not from a second list read off doc_matters. One list, in ssp.d: a property added there arrives in the database with nothing else changed, and one that is not in the .ssp cannot be in the database at all. That was the last place the two could drift; the objects stopped being able to on 2026-09-07. Version is checked on load. The major part must match, a newer minor is accepted (a minor bump only adds properties, and an unknown property line is ignored). A v0.1 artefact is now refused with a message saying to regenerate it, rather than loading half populated. test-abstraction-db.sh now compares the two artefacts' header blocks property by property, 36 per document on the wealth of networks, where it previously only checked that a schema.version row existed. Verified non-vacuous: dropping one property is reported. Reference regenerated, and the whole diff is this change and nothing else: 320 lines added, 35 removed over 35 files, being 35 format lines changed, 35 @source blocks (4 lines each), 35 language.document_char, 35 home_button_text (it has a default), 18 breaks, 16 footer, 4 date.added_to_site, 1 title.edition, 1 original.source. Existing outputs unaffected. (assisted by Claude-Code)
* ocda: .ssp should have all sections of abstractionRalph Amissah2026-09-121-1/+1
| | | | | | | The abstraction has nine sections and includes "tail". Both artefact writers hardcoded a list of eight, so "tail" was silently dropped. (assisted by Claude-Code)
* sqlite: schema version, & fail run on writes failRalph Amissah2026-09-091-4/+55
| | | | | | | | | | | | | | | | | | | | | | | | | Spine now declares sqlite_db_schema_version and stamps it into the database as PRAGMA user_version when the tables are created, in both the shared and the discrete DDL blocks. On opening an existing database it compares, and says once per run which version it found and which it writes. Failures are now tallied (shared, the output can run in parallel), reported one line each on stderr naming the operation, and main exits 1 without printing "run complete, ok". Two tests under test/, both taking the spine binary as their first argument and building their own database from data/pod unless $SpinePOD says otherwise: test-search-db-schema.sh names: every column the search form uses exists, and spine's declaration, the database's stamp and the search form's expectation all agree test-search-cgi.sh behaviour: the real search binary answers real requests against a fresh database, no web server involved, the probe values read out of whichever database it is given (assisted by Claude-Code)
* html metadata: a link to the ocda.dbRalph Amissah2026-09-091-4/+10
| | | | | | | The metadata page gains a line between the markup source and the source digests. (assisted by Claude-Code)
* --ocda-db replaces --show-abstraction-dbRalph Amissah2026-09-093-7/+25
| | | | | | | | | ocda (object centric document abstraction) flag --ocda-db (or --abstraction-db) replaces --show-abstraction-db rename results in consequently large diff
* output: the abstraction artefacts live with the podRalph Amissah2026-09-093-20/+81
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | pod/ holds all document source representations: pod/<doc>/ source tree pod/<doc>/media/abstraction/<uid>.ssp abstraction, as text pod/<doc>.zip tree, zipped, .ssp included pod/<doc>.digests.txt sha256s of what is in them pod/<uid>.ocda.db abstraction, as sqlite db <lang>/abstraction/ is gone. pod/<doc>/media/abstraction/<uid>.ssp preferred as having the images (found within the pod tree) which .ssp needs to reproduce a document but does not carry on its own. The .ocda.db sits carries the images as well and (like the pod.zip) can be used to reproduce a document directly. digests.txt now covers the database as well as the zip, the source and the .ssp; (as does the metadata html page). Two ordering issues addressed: - the pod builder clean-slates pod/<doc>/ before regenerating it, and the .ssp is now written before that runs. The clean slate now leaves media/abstraction/ alone, and the .ssp writer clears that directory itself on the first language of a run, so a .ssp for a language the document no longer is removed and cannot be bundled. - for a multi-language document the .ssp files accumulate one language at a time and are bundled on the last, which is why the directory cannot simply be emptied by whichever (language) gets there first. (assisted by Claude-Code)
* ocda db: <doc>.ocda.db, and a script to run 4 testsRalph Amissah2026-09-091-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The per document database is now written as <doc_uid>.ocda.db rather than <doc_uid>.abstraction.db. Shorter, and it says what is in the file: the object centric document abstraction, not "some abstraction". .ocda.db pairs with .ssp and cannot be mistaken for the collection search database (spine.search.db). test/run-tests.sh runs the four in sequence, one line of result each, and a summary. It re-runs itself inside nix shell "nixpkgs#sqlite" if sqlite3 is not on PATH, so this is all that is needed: SpinePOD=../../markup/sisudoc-spine-samples/markup/pod-samples/pod \ ./test/run-tests.sh ./bin/spine-ldc The tests are independent (with test-abstraction-ssp.sh run first): 1 test-abstraction-ssp.sh is first because it is the one that says whether the abstraction itself moved; if it fails the others are answering a different question than you think 2 test-abstraction-ssp-roundtrip.sh reads the committed reference set, so it is a statement about the current binary only if 1 passes 3 test-abstraction-db.sh and 4 test-abstraction-db-roundtrip.sh generate both artefacts themselves and depend on nothing committed (assisted by Claude-Code)
* ocda db: built from the .ssp, (tethered)Ralph Amissah2026-09-091-3/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The ocda database is now built from the .ssp itself: the writer's lines are emitted, read straight back by ssp_in, and the objects that come out populate the database. Anything the .ssp does not carry, the database will not have either, by construction (rather than by test). [instead of as previously through a second walk over the in-memory abstraction] - spineAbstractionTxt is split: sspDocumentLines(doc) returns the whole .ssp as lines, and the file writer emits them. Output neutral, the reference test confirms. - spineAbstractionDb takes the abstraction as an argument rather than taking doc.abstraction. - sspRoundTripAbstraction(doc) in ssp_in is the join: lines out, lines in, abstraction returned. Both call sites in spine.d use it. - the header blocks and the image blobs still come from doc_matters (as: the .ssp does not carry image bytes). All (35) markup sample sourced databases built through the .ssp have byte identical SQL dumps to the one built directly before the change. That comparison also found one reader inaccuracy, which the .ssp round trip could not see because the writer omits the field either way: an absent identifier was restored as the ocn in every case, but for an object with no ocn it was empty ("a"~N identifiers are always written). Fixed; the two artefacts checking each other is what caught it. (assisted by Claude-Code)
* .ssp: three additional fields the writers readRalph Amissah2026-09-091-0/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | With these the .ssp carries every field the output writers read. The two that were missing were both read by the sqlite search writer, and one of them is needed by anything that wants to know where an object sits in a segmented output: - .segment_html_is / .segment_epub_is (tags.html_segment_anchor_tag_is, epub_segment_anchor_tag_is), the segment an object *belongs to*, as against .segment (in_segment_html), which is the segment a heading *opens* and is set on headings only: 32 of 36 headings and 0 of 109 paragraphs in the autonomous contract. Without them a reader cannot tell which segment a paragraph is in. - .identifier for non-heading objects, the "a"~N series given to objects whose number is suppressed. A heading already carried it on its declaration line; every other object now carries it as a property, on the same condition, that it differs from the ocn. The same two segment fields are added to the abstraction db (segment_html_is, segment_epub_is) and to the cross-check test, so the two serialisations stay level. .ssp reference regenerated output run (--text --latex --epub --html --curate) byte identical to before the change, as it must be since no output writer reads either artefact. (assisted by Claude-Code)