aboutsummaryrefslogtreecommitdiffhomepage
path: root/test/reference
Commit message (Collapse)AuthorAgeFilesLines
* abstraction 2.1: a poem states its object rangeRalph Amissah2026-09-3036-439/+594
| | | | | | | | | | | | | | | - A poem is a container of verse, and stores its range of verse ocn, - its verse are the citable units with ocn. - A note in the last verse of a poem previously was not gathered into the endnotes section, this now is fixed Format 2.0 -> 2.1: the property is an addition, the poem is not a citable object (but contans a range of objects), its verse are (individual citable objects), and every reader checks the major version. A 2.0 database reads as having no ranges. (assisted by Claude-Code)
* notes: editor's notes, ~[* ]~ & ~[+ ]~, as seriesRalph Amissah2026-09-304-247/+1838
| | | | | | | | | | The markup documents ~[* note ]~ and ~[+ note ]~ as editor's notes, each a separately numbered series, and a bare ~[ note ]~ which sisu put in the asterisk series. They are included as notes. Each series is numbered through the document, *1, *2 ... and +1, +2 ..., apart from the author's notes. (assisted by Claude-Code)
* uid and paths separator "~" in place of ":"Ralph Amissah2026-09-231-0/+0
| | | | | | | | | | | | | | | | | | | need a character to split filenames on in certain circumstances. there problems with use of a colon in filenames, which is legal on posix but not on Windows."~" fits the bill better being legal on every filesystem of interest and is unreserved in rfc 3986, needing no escaping in a url. The reference abstraction is renamed, its content unchanged. Over the sample collection two filenames move and nothing else does: the abstraction and database digests are identical, the archive's members and their sizes are unchanged, and only the member order shifts, "~" collating after letters where ":" sorted before them. The document's epub dc:identifier is a v5 uuid derived from the uid, so it changes for those filenames. Every document's uid changes in the search database (spine.search.db). (assisted by Claude-Code)
* test: language alignment, (live-manual)Ralph Amissah2026-09-222-946/+954
| | | | | | | | | | | The ja and pl translations were mended in the sample set, so the two divergences this script asserted by name are gone. It failed and named every stale expectation, which is what it was for. The reference set is regenerated with two files affected: live-manual.ja.ssp and live-manual.pl.ssp. (assisted by Claude-Code)
* abstraction: format 2.0, source.digestsRalph Amissah2026-09-2236-72/+72
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | format 2.0, source.digest covers every markup file, the sha256 over one line per markup file, "<sha256> <filename>", sorted by filename. source.digest was the sha256 of the master file as read, before any insert. This is right for a .sst, which is the whole document. For a .ssm with inserts .ssi (containing the substantive part of a documents text) this is close to meaningless. Considered change of meaning (rather than an addition) and given the major part of the format version with it: 1.1 becomes 2.0, in the .ssp header line and in the database's schema.version row together. A 1.x reader refuses a 2.0 artefact, which is what that check is for. - Sorted, so the value does not depend on the order a filesystem hands back a directory. - By filename (rather than by path), so it does not depend on where the pod sits, which is the property the pod rebuild comparison rests on: a pod unzipped elsewhere must still give the same abstraction. Within one language every markup file lives in one directory, so a filename identifies it. Each line is checkable on its own against a digests.txt line or a files row, which the single value was not. The whole is reproducible with sha256sum and sort, and was verified that way for a one file document and for a twenty file one. An unreadable file is named in the digest rather than skipped. The parse has failed elsewhere by then, and a digest that quietly left a file out would claim the document is something it is not. The reference abstractions are regenerated. Two lines change in each and no others: the version, and the digest. (assisted by Claude-Code)
* test (ref-abstr-ssp): live-manual mended headersRalph Amissah2026-09-229-19/+19
| | | | | | | | | | | | | | | | reference abstraction follows live-manual's mended headers The sample's catalogues were regenerated and its nine translated headers mended, so those nine abstractions change. Two lines each for ca and es, which is the source digest alone and no metadata at all: their headers came back exactly as they were. Four to six for the rest, where a copyright line drops its "Copyright (C)" prefix to follow the English, and where de, pt_BR and ro gain a licence in their own language. live-manual.en.ssp is unchanged, being the language translated from, and so are the other twenty seven documents, byte for byte. (assisted by Claude-Code)
* test (ref-abstr-ssp): live-manual + pt_BRRalph Amissah2026-09-2210-9/+16040
| | | | | | | | | | The sample pod now lists pt_BR, so the document has a tenth language. No other document's reference is touched; the other twenty six are byte identical. (assisted by Claude-Code)
* images alt text, and epub says what it isRalph Amissah2026-09-142-26/+26
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Four things, not visible in an error count: - Images had no alt text, even where markup carried the text, and the field alt text is present, incorrectly used in code; fixed as the alt. { sm_tux.png 64x80 "Gnu/Linux - a better way" }image an image with nothing to say for itself gets alt="" - Images with no dimensions were incorrectly written width="0" height="0", fixed - Accessibility metadata, which EPUB Accessibility 1.1 requires: schema:accessMode, accessibilityFeature and accessibilityHazard, plus accessModeSufficient and accessibilitySummary, which epubcheck does not test for; used by - readers looking for a book it can use, - anyone distributing into the EU. Every value is derived from the document, not asserted: a document with no images says so, one whose images all carry alt text claims alternativeText and textual sufficiency, one where an image says nothing claims neither. Nothing claims WCAG conformance, which would be a claim about an evaluation that has not happened. - DPUB-ARIA roles beside the structure spine already knew about. doc-toc, doc-endnotes, doc-bibliography, doc-glossary and doc-index on the sections, doc-noteref on a note reference and doc-footnote on the note it points at. "doc_endnotes" as a class name means something to a stylesheet and nothing to a reading system. Not doc-biblioentry: DPUB- ARIA 1.1 deprecated it. These are plain ARIA, so the html gets them too. Also dealt with here: - Spine makes certain sections such as endnotes, bookindex, ... and a heading that has a name for itself cannot share the same name, if it does the name is now disambiguated, disambiguated. (disambiguation similar to that used for a repeated anchor tag) - Endnote anchors fixed to use "id" (being one unique tag in the document that is target of every reference to it)instead of obsolete form (they were <a name="note_1">. "name" on an <a>). Book index markers keep "name": the same marker sits on every object an entry names, 296 times in one document, and those are markers of where a term occurs rather than distinct targets. (assisted by Claude-Code)
* epub and html: no errors left over the sample setRalph Amissah2026-09-143-25/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | epubcheck 5.3.0 over the 35 sample epubs went from 1302 remaining errors to none. The six causes for those remaining errors fixed here. most of the count was one of them: - Tables (1123 errors, 86%) where the writer emitted obsolete attributes, removed in 2014; table also now in div instead of a paragraph. also fixed: - Navigation (115). Every level 4 heading's nav entry ended "#0", the ocn of an object that has no ocn and so no id. - Duplicate ids (37). (from three unrelated sources). - The publication identifier (27 warnings). dc:identifier was a hardcoded hex string, not a UUID, and the same one in all 35 files, so two documents in one library collided. It is now UUID v5 over the document's own uid: a real RFC 4122 identifier, stable across builds and distinct per document. - Image manifest entries (15). media-type was "image/" plus the file extension, giving "image/jpg", which is not a media type; epubcheck reads it as a foreign resource and wants a fallback. There is now a mapping, which also covers svg and webp. Item ids came from the file basename, and "2bits_02_01-100.png" gave an id starting with a digit, which is not an XML name; ids are now prefixed and sanitised. - An ocn resolving to no segment (9). A poem block and its first verse share one ocn; the verses are written into a segment and the block itself is not. Also dropped from the package document: an xmlns:xsi that nothing used, and a prefix declaration for the rendition vocabulary, which is a reserved prefix and needed no declaring. The package now carries xml:lang. test-epub-validity.sh now allows no fatal and no error. Two NAV-011 warnings remain and are described in it. Three reference .ssp files change, all of them the anchor tag disambiguation above. (assisted by Claude-Code)
* regex: confine url within an endnote (delimiter)Ralph Amissah2026-09-141-3/+3
| | | | | | | in error regex permitted a url to go past an endnote close delimiter, fixed (assisted by Claude-Code)
* ocda: heading cross reference linkingRalph Amissah2026-09-1235-0/+317
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The segment a cross reference to a heading lands on, gixed upstream in ocda, where the field is created, rather than inferred by the loader. A heading above level 4 opens no html segment of its own, so a link to it has to land on the level 4 heading that follows. The build loop cannot know that when it reads the heading, and says so in a comment of its own: "for html segname need following lv4 not yet known". It back-fills tag_assoc when the level 4 heading arrives (lv0to3_tags), and the answer then lives only in that map. tags.segment_lv4_is now holds it, resolved in a pass over the finished head and body sections: walking backwards, the last level 4 heading seen is the next one for everything above it, which is the same answer the back-fill gives. Emitted in the .ssp only where it differs from .segment_html_is, which is every heading at level 4 or below, so it is sparse: 317 lines over the 35 document reference set, none removed. Carried in the database as a column of its own, and read back by both readers. docHasFromAbstraction reads it instead of working it out. With that, and on top of the two defect fixes, a document loaded from an artefact and one parsed from markup agree: key sets identical, all 35 documents values 3 entries differ of some 30,000, all of them _the_title, where the abstraction supplies an epub segment the parser leaves unset link targets all 1,651 agree, against 89 differing before any of this work and 1 after the defect fixes alone Format stays v1.1. That version is new in this same run of work and nothing outside spine has read it, so this belongs in it rather than in a bump of its own. (assisted by Claude-Code)
* ocda: the abstraction carries the document metadata output readsRalph Amissah2026-09-1235-35/+355
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | .ssp format v0.1 -> v1.1, and the .ocda.db with it: they are two serialisations of one format and now move together. The gap this closes, measured against the writers' read set: of the 59 conf_make_meta fields outputs/io_out/ reads, 29 were in neither artefact. Five are conf.* and stay out, being site and run scoped (urls, papersize, the search db filename): the same abstraction published to two sites must take each site's. Of the rest, three are never assigned anywhere (title_short, publisher, original_publisher; read only by sqlite.d, always empty) and one is a copy of a field already carried (title_sub = title_subtitle), so 17 properties actually had to travel and now do: make breaks, footer, home_button_text meta title.edition, date.added_to_site, language.document_char, original.{title,source,language,language_char}, rights.copyright_{text,translation,illustrations, photographs,cover,audio,video} make splits by when it acts, which is worth keeping in mind: these three are read in outputs/io_out/ and must travel, while italics, bold, emphasis, substitute and headings are read at parse time and their effect is already in the objects. New @source block, and source.* rows in the database, so a reader can say which markup an abstraction came from rather than working from stale content in silence: language the document's own languages the pod's list, which is what the inter-language links in html need and neither artefact carried digest sha256 of the .sst; equals its digests.txt entry The database adds source.ssp_digest, the sha256 of the .ssp it was built from, since a file cannot hold its own hash. So the chain .sst -> .ssp -> .ocda.db is checkable end to end. The database's metadata table is now filled from the header blocks the .ssp gives back, not from a second list read off doc_matters. One list, in ssp.d: a property added there arrives in the database with nothing else changed, and one that is not in the .ssp cannot be in the database at all. That was the last place the two could drift; the objects stopped being able to on 2026-09-07. Version is checked on load. The major part must match, a newer minor is accepted (a minor bump only adds properties, and an unknown property line is ignored). A v0.1 artefact is now refused with a message saying to regenerate it, rather than loading half populated. test-abstraction-db.sh now compares the two artefacts' header blocks property by property, 36 per document on the wealth of networks, where it previously only checked that a schema.version row existed. Verified non-vacuous: dropping one property is reported. Reference regenerated, and the whole diff is this change and nothing else: 320 lines added, 35 removed over 35 files, being 35 format lines changed, 35 @source blocks (4 lines each), 35 language.document_char, 35 home_button_text (it has a default), 18 breaks, 16 footer, 4 date.added_to_site, 1 title.edition, 1 original.source. Existing outputs unaffected. (assisted by Claude-Code)
* ocda: .ssp should have all sections of abstractionRalph Amissah2026-09-1235-8/+498
| | | | | | | The abstraction has nine sections and includes "tail". Both artefact writers hardcoded a list of eight, so "tail" was silently dropped. (assisted by Claude-Code)
* run abstraction test re-generate reference .sspRalph Amissah2026-09-0928-1789/+1541
|
* .ssp: three additional fields the writers readRalph Amissah2026-09-0935-0/+92359
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | With these the .ssp carries every field the output writers read. The two that were missing were both read by the sqlite search writer, and one of them is needed by anything that wants to know where an object sits in a segmented output: - .segment_html_is / .segment_epub_is (tags.html_segment_anchor_tag_is, epub_segment_anchor_tag_is), the segment an object *belongs to*, as against .segment (in_segment_html), which is the segment a heading *opens* and is set on headings only: 32 of 36 headings and 0 of 109 paragraphs in the autonomous contract. Without them a reader cannot tell which segment a paragraph is in. - .identifier for non-heading objects, the "a"~N series given to objects whose number is suppressed. A heading already carried it on its declaration line; every other object now carries it as a property, on the same condition, that it differs from the ocn. The same two segment fields are added to the abstraction db (segment_html_is, segment_epub_is) and to the cross-check test, so the two serialisations stay level. .ssp reference regenerated output run (--text --latex --epub --html --curate) byte identical to before the change, as it must be since no output writer reads either artefact. (assisted by Claude-Code)
* sisu markup sample .sst editsRalph Amissah2026-09-091-2329/+2778
| | | | track on sisu markup sample
* test-abstraction-ssp re-generatedRalph Amissah2026-09-0335-42562/+9614
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Several structure fields (now) computed in ocda do not yet have a consumer outside the `.ssp` writer (which shares ocda fields in a PEG parsable .ssp file). Their addition is to facilitate future output processing: each representing document structure information that document output generators of various formats, such as html, epub, latex, sql and any DOM based / tree-graph (node based hierarchical) representations, previously had to, or would have to, work out for themselves based on what ocda offered (without them). Computing them (upstream) in ocda once and sharing them (and providing the .ssp), means that: - future output formats can start from prepared structure rather than deriving it; - existing generators can be simplified to consume them instead of re-deriving; - a (dlang) generator could in principle be populated either from ocda directly or from a `.ssp` file; - `.ssp` being PEG parsable puts the same prepared structure in reach of implementations in other languages. `test/test-abstraction-ssp.sh` the `.ssp` reference regression test is used to catch regression in ocda including these as yet unused fields, which otherwise would remain unseen until used by other output-format consumers. This also is the reason issues now made easily visible and are being addressed (without issue to existing output generators). (assisted by Claude-Code)
* test-abstraction-ssp re-generated, abstraction fixRalph Amissah2026-08-3035-5737/+197037
| | | | | test-abstraction-ssp re-generated: document abstraction (ocda) document structure representation fixes (verify)
* .ssp: doc structure related fixes (& to epub toc_nav)Ralph Amissah2026-08-281-40/+40
| | | | | | | | | | | | | | | | | | | | | | | | | | for document abstraction and its .ssp output, removed the requirement of including --abstraction & --serial flags to produce correct output (for: .dom_status, .dom_status_collapsed & .last_descendant) - meta_processing_xml_dom() includes show_abstraction, so --pod2 and --show-abstraction run the dom pass; last_descendant is derived from that pass via after_doc_get_descendants() The accumulators are now verified as eight wide locals of docAbstraction(), so each document starts clean and no two threads share one. - bug: the four dom accumulators were template scope (shared) and nine wide, while their end of document reset was eight wide, so the first document of a run differed from the rest and parallel runs raced on one buffer, (which also mis-nested epub toc_nav) test/ reference .ssp regenerated: accelerando only, trailing zero dropped. test-abstraction-ssp.sh now runs parallel and diffs output against a serial run. (assisted by Claude-Code)
* test build .ssp requires --serial (--no-parallel) runRalph Amissah2026-08-2816-2101/+2101
|
* test build .ssp requires --abstraction to be runRalph Amissah2026-08-2835-152/+9218
|
* run test-abstraction-ssp.sh updateRalph Amissah2026-08-2830-106/+171
| | | | | generrate new reference set of .ssp test-abstraction-ssp.sh --generate
* digest (sha256) for document root heading (title)Ralph Amissah2026-07-3135-0/+35
| | | | | | corrects an omission this hould have been included to start with reference tests texts regenerated
* ssp test-abstraction-ssp.sh script minorRalph Amissah2026-05-152-4/+4
| | | | | | | | | - test/test-abstraction-ssp.sh script minor improvement - result change reflects content corrections: name spelling & a markup error, document abstracton, ssp verified ok (may be considered a test) (assisted by Claude-Code)
* add .ssp regression test script and reference filesRalph Amissah2026-04-2235-0/+444119
test/test-abstraction-ssp.sh: shell script that generates .ssp files for all sample documents and diffs against committed reference files in test/reference/abstraction/. Usage: ./test/test-abstraction-ssp.sh --generate ./bin/spine-ldc ./test/test-abstraction-ssp.sh ./bin/spine-ldc Exits 0 if all .ssp files match, 1 if differences found. Reports changed, missing, and new files. test/reference/abstraction/: 35 reference .ssp files generated from the current abstraction, covering all sample documents including 9-language live-manual. Co-Authored-By: Anthropic Claude Opus 4.6 (1M context)