| Commit message (Collapse) | Author | Age | Files | Lines |
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Check that they are the right files: source.digest is the sha256 over
one line per markup file, "<SHA256> <basename>", sorted and newline
joined, so the same value can be computed from the stored rows alone and
held against what the .ssp recorded. A file stored under the right name
with the wrong contents passes a count and fails this.
Two details the shell gets wrong by default: the sort has to be
LC_ALL=C, because the writer sorts by byte and a locale does not, and
the lines are joined with no trailing newline, so printf and not echo.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Asserts the two known divergences in live-manual's translations by name
and value: ja two objects short of en, and pl's ocn 313 a paragraph in
English and a heading in Polish.
It asserts that nothing else is reported: exactly one document warned
about, exactly two languages of it.
Also the exit codes: 0 without --strict, non-zero with it on the
diverging document, and 0 with it on a single language document.
It will fail when those two translations are mended, which is the right
moment to hear about it, and the message names the expectation that went
stale.
test-abstraction-db.sh excludes translation.ocn_aligned from its header
block comparison, with the other two keys that are the database's notes
about the artefact rather than properties of the document.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
one file holding several languages.
Both assumed <uid>.<lang>.ocda.db beside <uid>.<lang>.ssp, one for
one.
- test-abstraction-db.sh takes the language from the .ssp name, the
database from the stem, and scopes every query by the doc_id that
language resolves to: objects by doc_id, the four child tables
through their objects, metadata by doc_id, schema.version by doc_id
is null. It also checks what a per language loop cannot see, that the
languages a database holds are exactly the languages there are .ssp
files for.
- test-abstraction-db-roundtrip.sh asks for each language by name and
holds it against its own .ssp, and requires a database of more than
one language to refuse a round trip that names none: a reader quietly
emitting the first language would otherwise pass against the first
.ssp and say nothing about the other nine. Its language list comes
from the .ssp filenames, so it still needs no sqlite3.
databases (re-)emitting all .ssp files exactly.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
.ssp format v0.1 -> v1.1, and the .ocda.db with it: they are two
serialisations of one format and now move together.
The gap this closes, measured against the writers' read set: of the 59
conf_make_meta fields outputs/io_out/ reads, 29 were in neither
artefact. Five are conf.* and stay out, being site and run scoped (urls,
papersize, the search db filename): the same abstraction published to
two sites must take each site's. Of the rest, three are never assigned
anywhere (title_short, publisher, original_publisher; read only by
sqlite.d, always empty) and one is a copy of a field already carried
(title_sub = title_subtitle), so 17 properties actually had to travel
and now do:
make breaks, footer, home_button_text
meta title.edition, date.added_to_site, language.document_char,
original.{title,source,language,language_char},
rights.copyright_{text,translation,illustrations,
photographs,cover,audio,video}
make splits by when it acts, which is worth keeping in mind: these three
are read in outputs/io_out/ and must travel, while italics, bold,
emphasis, substitute and headings are read at parse time and their
effect is already in the objects.
New @source block, and source.* rows in the database, so a reader can
say which markup an abstraction came from rather than working from
stale content in silence:
language the document's own
languages the pod's list, which is what the inter-language links
in html need and neither artefact carried
digest sha256 of the .sst; equals its digests.txt entry
The database adds source.ssp_digest, the sha256 of the .ssp it was built
from, since a file cannot hold its own hash. So the chain
.sst -> .ssp -> .ocda.db is checkable end to end.
The database's metadata table is now filled from the header blocks the
.ssp gives back, not from a second list read off doc_matters. One list,
in ssp.d: a property added there arrives in the database with nothing
else changed, and one that is not in the .ssp cannot be in the database
at all. That was the last place the two could drift; the objects stopped
being able to on 2026-09-07.
Version is checked on load. The major part must match, a newer minor is
accepted (a minor bump only adds properties, and an unknown property
line is ignored). A v0.1 artefact is now refused with a message saying
to regenerate it, rather than loading half populated.
test-abstraction-db.sh now compares the two artefacts' header blocks
property by property, 36 per document on the wealth of networks, where
it previously only checked that a schema.version row existed. Verified
non-vacuous: dropping one property is reported.
Reference regenerated, and the whole diff is this change and nothing
else: 320 lines added, 35 removed over 35 files, being 35 format lines
changed, 35 @source blocks (4 lines each), 35 language.document_char, 35
home_button_text (it has a default), 18 breaks, 16 footer,
4 date.added_to_site, 1 title.edition, 1 original.source.
Existing outputs unaffected.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
| |
ocda (object centric document abstraction)
flag --ocda-db (or --abstraction-db)
replaces --show-abstraction-db
rename results in consequently large diff
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
pod/ holds all document source representations:
pod/<doc>/ source tree
pod/<doc>/media/abstraction/<uid>.ssp abstraction, as text
pod/<doc>.zip tree, zipped, .ssp included
pod/<doc>.digests.txt sha256s of what is in them
pod/<uid>.ocda.db abstraction, as sqlite db
<lang>/abstraction/ is gone.
pod/<doc>/media/abstraction/<uid>.ssp preferred as having the
images (found within the pod tree) which .ssp needs to reproduce a
document but does not carry on its own.
The .ocda.db sits carries the images as well and (like the
pod.zip) can be used to reproduce a document directly.
digests.txt now covers the database as well as the zip, the source
and the .ssp; (as does the metadata html page).
Two ordering issues addressed:
- the pod builder clean-slates pod/<doc>/ before regenerating it,
and the .ssp is now written before that runs. The clean slate
now leaves media/abstraction/ alone, and the .ssp writer clears
that directory itself on the first language of a run, so a .ssp
for a language the document no longer is removed and cannot be
bundled.
- for a multi-language document the .ssp files accumulate one
language at a time and are bundled on the last, which is why the
directory cannot simply be emptied by whichever (language) gets
there first.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The per document database is now written as <doc_uid>.ocda.db
rather than <doc_uid>.abstraction.db. Shorter, and it says what is
in the file: the object centric document abstraction, not "some
abstraction". .ocda.db pairs with .ssp and cannot be mistaken for
the collection search database (spine.search.db).
test/run-tests.sh runs the four in sequence, one line of result
each, and a summary. It re-runs itself inside nix shell
"nixpkgs#sqlite" if sqlite3 is not on PATH, so this is all that
is needed:
SpinePOD=../../markup/sisudoc-spine-samples/markup/pod-samples/pod \
./test/run-tests.sh ./bin/spine-ldc
The tests are independent (with test-abstraction-ssp.sh run
first):
1 test-abstraction-ssp.sh is first because it is the
one that says whether the abstraction itself moved; if it
fails the others are answering a different question than you
think
2 test-abstraction-ssp-roundtrip.sh reads the committed
reference set, so it is a statement about the current binary
only if 1 passes
3 test-abstraction-db.sh and
4 test-abstraction-db-roundtrip.sh generate both artefacts
themselves and depend on nothing committed
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
With these the .ssp carries every field the output writers read.
The two that were missing were both read by the sqlite search
writer, and one of them is needed by anything that wants to know
where an object sits in a segmented output:
- .segment_html_is / .segment_epub_is
(tags.html_segment_anchor_tag_is, epub_segment_anchor_tag_is),
the segment an object *belongs to*, as against .segment
(in_segment_html), which is the segment a heading *opens* and is
set on headings only: 32 of 36 headings and 0 of 109 paragraphs
in the autonomous contract. Without them a reader cannot tell
which segment a paragraph is in.
- .identifier for non-heading objects, the "a"~N series given to
objects whose number is suppressed. A heading already carried it
on its declaration line; every other object now carries it as a
property, on the same condition, that it differs from the ocn.
The same two segment fields are added to the abstraction db
(segment_html_is, segment_epub_is) and to the cross-check test, so the
two serialisations stay level.
.ssp reference regenerated
output run (--text --latex --epub --html --curate) byte identical
to before the change, as it must be since no output writer reads
either artefact.
(assisted by Claude-Code)
|
|
|
cross-check .ssp & ocda (abstraction) db against each other
To prevent drift, this compares them directly to detect one writer
gaining a field and the other not (rather than each against a
committed reference, so there is no new fixture to keep current)
For every sample document it generates both artefacts and asserts:
- the same number of objects, and the same number in each of the eight
sections
- for 32 single valued properties, that the count of the .ssp property
equals the count of rows where the corresponding column is set
- for the four repeatable properties (.image, .stow_link, .anchor_tag,
.lev4_subtoc), that the count equals the rows in its table
- that every image the abstraction referred to and found is carried in
the files table
- that the text of the first and last citable body object is byte
identical in the two
- that the database says what it is (metadata schema.version)
SpinePOD=../../markup/sisudoc-spine-samples/markup/pod-samples/pod \
nix shell "nixpkgs#sqlite" -c ./test/test-abstraction-db.sh ./bin/spine-ldc
A sqlite error on a missing column is reported as a mismatch
rather than aborting the run.
The other half of this step, one shared field description behind
both writers, is not done here: the cross-check gives the same
guarantee without touching either writer, and the question becomes
moot if, as planned, the database comes to be generated from the
.ssp through the loader.
Added with -f: test/ is covered by .gitignore's catch-all, as
test/test-abstraction-ssp.sh was before it.
(assisted by Claude-Code)
|