aboutsummaryrefslogtreecommitdiffhomepage
path: root/src/sisudoc/outputs/io_out/source_pod.d
Commit message (Collapse)AuthorAgeFilesLines
* cleanup: misc. (previously deferred)Ralph Amissah2026-09-221-3/+9
| | | | | | | | | | | | | | | | | | | | | dr_document_make in three identifier spellings becomes document_make, which is what the file has been called since the rename. The mixin import audit, a template declares its imports at template scope and they are then visible in every scope it is mixed into (which is how a template-level split once hijacked a UFCS lookup in code that had not been touched). Disabling htmlSnippet's imports and rebuilding names the consumers that were relying on them rather than on their own: across twelve mixin sites, exactly one. metadata.d now imports the `to` it uses. The templates keep their imports, their own functions needing them, and narrowing those is a change of its own rather than a cleanup. And the standing FIX in source_pod.d: the insert digest line for a non pod source recorded the insert's path, where every other line in digests.txt is a filename and a digest naming a file that is not in the pod under that name cannot be checked against it. (assisted by Claude-Code)
* digests: db digest taken once on file completionRalph Amissah2026-09-221-16/+19
| | | | | | | | | | One file for the document (including all languages) with digest of the document. digests.txt lists it once, beside the shared images, taken when the last language has been written. digests.txt carries all digests. (assisted by Claude-Code)
* pod: carry tools/po4a, the translation working setRalph Amissah2026-09-221-0/+41
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A pod now carries its catalogues, so a translation can be updated rather than only retyped: providing the .pot and .po (the means of maintaining it). A directory walk, where every other thing the pod writer carries is named. A catalogue set is whatever the translator has, and enumerating .pot and .po files would mean the writer deriving the document's languages a second time, from a second place, to say what it already reads from the manifest. Done once, on the last language, as the markup blocks are. The tree is not language specific, so a pass for each language would re-read all of it, 5.5 MB ten times over for live-manual, into an archive the next pass overwrites. Every name is checked before it is carried, by the rule the pod reader applies to an archive it is handed. The writer is reading a directory someone else may have filled, which is the reason the reader checks. The cost is small, and it is the whole argument for compressing the archive as a stream rather than per member: uncompressed content 6972418 -> 13167489 +89% the .sisupod 692942 -> 891388 +28.6% 5.5 MB of catalogues cost 198 KB, because the .po files largely repeat markup already in the archive and whole-stream compression sees it. Per-member deflate could not. (assisted by Claude-Code)
* pod: set entry limit for the writer and readerRalph Amissah2026-09-221-0/+38
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Both now take the number from one constant now set at 10,000, the writer checks it before the archive is written. Previously the number set for the pod reader was less than 500 members. The writer had no limit, (so spine could write a pod it would then refuse to read, reporting too many entries: a good file that looks corrupt, and only for a document with enough parts). 500 was set when a pod held markup and images. A pod carrying translation catalogues has a different arithmetic, and the count follows from how many files a document is made of times how many languages it has, not from how large the document is: the_wealth_of_networks 213,405 words, 1 file per language 12 languages -> 49 entries live-manual 24,724 words, 20 files per language 12 languages -> 506 entries The big book is not what runs into this; the modular manual is. live-manual stood two languages from an unreadable artefact. 10,000 covers a hundred-insert manual in thirty languages, about 6,200 entries, with room. It is deliberately well clear of any real document rather than snug above the largest one known: the two errors are not comparable, since too low refuses a legitimate document with a message that reads as corruption, while too high defers to a size cap a moment later. The count is the weakest of the three guards and is not what bounds resource use; the per entry and total size caps do that, both before a byte is written. Refusing also removes any archive an earlier language left. The writer runs once per language and only the last pass holds every language, so it is the last that goes over, and the passes before it wrote smaller archives that passed. Without this, refusal left a pod missing a language: an artefact that reads perfectly well and is wrong. A missing file is an error someone notices. The arithmetic is recorded beside the constant so the next person can re-derive the number rather than guess at it. (assisted by Claude-Code)
* pod: write <doc>.sisupod (single zstd frame)Ralph Amissah2026-09-221-4/+21
| | | | | | | | | | | | | | | | | | | | | | | | | | | | The pod archive spine writes is now named .sisupod and carries the same archive inside one zstd frame. The reader has accepted both since the previous commit, so every pod already published stays readable and this changes only what is written. free_culture 2863504 -> 1100180 live-manual 6972418 -> 692942 10.06x The wrap is at the pod call site and not in createZipFile, which also writes every epub and every odt. Best compression from whole-stream (rather than per-member compression). (for live-manual per-member deflate manages about 3x compared to 10x for whole pod content compression). The suffix is named as pod_archive_suffix. --pod-compression sets the level, 19 by default: a pod is written once and fetched many times (692942 bytes at 19, 837703 at 9, 2054365 at 1). (A value that is not a number warns and the default is used). Output built from a written .sisupod is byte identical to output built from the markup it came from. (assisted by Claude-Code)
* conf: read conf/document_make, drop the dr_ prefixRalph Amissah2026-09-221-4/+4
| | | | | | | Spine now looks for conf/document_make, and bundles that name. (the dr_ prefix was a sisu-era name to distinguish it) (assisted by Claude-Code)
* pod digests: multilingual document digests sortedRalph Amissah2026-09-221-7/+16
| | | | | | | | | | | | | | Fixes for three faults in the insert (.ssi) digests. The key was the language captured from whichever path resolved the insert. It did not change across the loop over languages, so every language's inserts were filed under one language. The key is now the language being digested, as the digest of the primary file above it already did. - digest is taken after the check that the file exists. - digest is now taken inside the guard. - non-pod path now correctly keyed on the document's own language. (assisted by Claude-Code)
* some file renamesRalph Amissah2026-09-121-1/+1
|
* --ocda-db replaces --show-abstraction-dbRalph Amissah2026-09-091-5/+23
| | | | | | | | | ocda (object centric document abstraction) flag --ocda-db (or --abstraction-db) replaces --show-abstraction-db rename results in consequently large diff
* output: the abstraction artefacts live with the podRalph Amissah2026-09-091-15/+53
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | pod/ holds all document source representations: pod/<doc>/ source tree pod/<doc>/media/abstraction/<uid>.ssp abstraction, as text pod/<doc>.zip tree, zipped, .ssp included pod/<doc>.digests.txt sha256s of what is in them pod/<uid>.ocda.db abstraction, as sqlite db <lang>/abstraction/ is gone. pod/<doc>/media/abstraction/<uid>.ssp preferred as having the images (found within the pod tree) which .ssp needs to reproduce a document but does not carry on its own. The .ocda.db sits carries the images as well and (like the pod.zip) can be used to reproduce a document directly. digests.txt now covers the database as well as the zip, the source and the .ssp; (as does the metadata html page). Two ordering issues addressed: - the pod builder clean-slates pod/<doc>/ before regenerating it, and the .ssp is now written before that runs. The clean slate now leaves media/abstraction/ alone, and the .ssp writer clears that directory itself on the first language of a run, so a .ssp for a language the document no longer is removed and cannot be bundled. - for a multi-language document the .ssp files accumulate one language at a time and are bundled on the last, which is why the directory cannot simply be emptied by whichever (language) gets there first. (assisted by Claude-Code)
* digests (sha256) a revisitRalph Amissah2026-07-311-15/+27
|
* ocda + outputs split: module/import + dub.json fixupsRalph Amissah2026-05-251-8/+8
| | | | | | | | | | | Modules and imports rewritten to sisudoc.ocda.* and sisudoc.outputs.*; dub.json excludedSourceFiles and the spine:abstraction sub-package sourcePaths collapsed to ./src/sisudoc/ocda. Verified: nix build .#spine-overlay-ldc clean. (assisted by Claude-Code)
* separate abstraction lib from output processingRalph Amissah2026-05-251-0/+544
create new directories under ./src/sisudoc ocda & outputs in order to separate the document abstraction library from downstream output processing (stuff broken till paths & modules fixed)