| Commit message (Collapse) | Author | Age | Files | Lines |
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- A poem is a container of verse, and stores its range of verse ocn,
- its verse are the citable units with ocn.
- A note in the last verse of a poem previously was not gathered into
the endnotes section, this now is fixed
Format 2.0 -> 2.1: the property is an addition, the poem is not a
citable object (but contans a range of objects), its verse are
(individual citable objects), and every reader checks the major version.
A 2.0 database reads as having no ranges.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
does this spine still produce this abstraction?
A database states an abstraction and carries the markup it was built
from, so re-parsing the one and comparing against the other says whether
the two still agree. Checked as follows:
- automatically and unskippably when a document is built from a
database, at the moment it is parsed and before anything has been
written from it.
- on --ocda-verify=<file>, an action of its own that materialises,
parses every language, compares, reports and writes nothing, the exit
status being the answer.
- on --no-verify as the escape, warning per document, for a newer spine
reading an older artefact where the abstraction is expected to differ
and the output is wanted anyway.
A build ends at the first failure rather than skipping the language.
A document rebuilt from a database now has its whole abstraction built.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The pod carries tools/po4a and the database (until now) did not, so a
pod written back out of a database came back a lossy copy, without its
translation catalogues.
the database now carries them, walked and name-checked exactly as the
pod writer walks and checks them, once on the last language. They are
compressed to save space, the test sample carrying 6.25 MB of catalogue
as read would have made that one document's database two thirds larger.
Files gains a compression column: NULL or 'none' for the bytes as read,
'zstd' for a frame. What to compress is decided by role, not by size
(source, conf, manifest and tools are text and are compressed; images
being compressed already are not). The same kind of file is always
stored the same way. Document objects are untouched and stay raw:
objects_fts is external content over objects.text and reads that column
directly.
bytes and sha256 remain those of the original file, so every digest
check, digests.txt line and source.digest rebuilt from these rows
works unchanged, and a reader that does not decompress can still say
what it is looking at.
dbReadFiles decompresses, so no caller learns how a blob is stored. It
asks pragma_table_info whether the column is there at all: a database
written before it reads as raw. A row that claims zstd and will not
decompress is named and left out, rather than handed back as a frame
where markup should be.
live-manual: 9,490,432 bytes without the catalogues, 9,908,224 with
them. Output from the materialised pod is identical to output from the
original pod, 679 files each side, no differences.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The database already carried the images, because without them no output
can be produced. It now carries the markup as well, and with it the
document's configuration and the manifest as the author wrote it.
That is the difference between a serialised abstraction and a document
source. An abstraction can be rendered but not re-parsed, and a reader
who wants to correct a sentence needs the sentence as written. With
these a pod can be written back out of the sqlite-file.
Three new values in the existing role column, no schema change: the
format stays 2.0.
Source rows are named by their path within the pod,
media/text/<lang>/<file>, and not by bare filename as images are. Every
language of a document has a file of the same name, so bare names would
collide under UNIQUE(role, name) and nine of ten would be dropped
without a word. The path is also what a pod materialised from this
database has to be told.
Bytes stored as read with the digest over them, so that each row is
checkable against the line source.digest was built from.
Also build.spine_version, a file level row saying which spine wrote the
file: not a property of the document, and the one thing a file cannot be
asked for afterwards. The reader skips the build. prefix as it skips
schema. and translation., or it would come back inside the document
header and the round trip would differ.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The check runs once every language of a document has been abstracted.
Providing a warning by default. --strict makes divergence a failure,
taken at the end of the run so that outputs are complete and can be
examined. (the name --strict is general, so that later checks can be
added without a second flag).
Under --parallel the profiles are appended under synchronized, and the
comparison sorts a document's languages by name rather than taking
them in the order the threads finished, so two runs print the same
lines in the same order.
The outcome is noted per language in the database as
translation.ocn_aligned, for documents that have more than one
language.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A database now holds every language of its document, so reading one
means naming a language. Three small pieces:
- spineDbLanguages says what languages a file holds. Its own template
rather than part of the reader (asking a file what it holds does not
instantiate the object setter and the markup regexes with it).
- abstractionLoad and spineDocFromArtefact take a language and pass it
to dbReadFile, which resolves it to a doc_id, takes that language
document where there is one, and lists the languages and stops where
there is a choice.
- spineArtefactLanguages turns an artefact into the documents to build:
one for a .ssp, and for a database the languages it holds filtered by
--lang. The artefact loop then has a language loop inside it, so a
single .ocda.db argument builds every language, as the pod it came
from would.
--db-round-trip takes --lang for the same reason, and ocda-db looses
--parallel (would cause a race).
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
each file still holds one language, here groundwork for one database per
document. The shape changes here and nothing merges yet: a database is
still written per language, so every doc_id is 1. Real and testable on
its own, where the writer and the reader together are not.
documents, a row per language, is what lets one file hold a document's
whole set. Everything that tells one language's rows from another's keys
on documents.id.
metadata is keyed on (doc_id, key) (no longer on key alone). Every
language has a title and a creator, and a key-only primary key refuses
the second one. schema.name and schema.version describe the file rather
than a document in it, so they are written with a null doc_id and are
the only rows that are.
objects gain doc_id and its uniqueness widens from (section, seq) to
(doc_id, section, seq). ('body', 0) exists once per language, so the
narrow constraint was the thing that would have refused a second
language outright. idx_objects_section leads with doc_id, or reading one
language scans them all.
objects.id stays a global INTEGER PRIMARY KEY, so object_images,
object_links, object_anchors and object_subtoc keep their schema and
their keys, and objects_fts keeps content_rowid='id'. The reader's four
sweeps filter through objects rather than gaining a column of their own.
outline and citable name the language and order by it first. A view over
a file that may hold several languages and does not say which reads as
one document and is several.
The DDL is IF NOT EXISTS throughout, since a second language will open a
file that already has its schema.
dbReadFile takes an optional language and means "the only document in
it" without one. Given none where there are several it reports the
languages and stops, rather than returning the first: the round trip
compares byte for byte, and a quietly wrong answer there would read as a
spine fault.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The document markup sample collection builds byte identically from
either artefact: 1457 files, no differences, from markup, from the 35
.ssp files, and from the 35 .ocda.db files.
A .ocda.db carries its images as blobs. dbReadFiles() reads them and
they are written where the output writers look for images: a
directory of this run's making, removed when the run ends, not the
pod beside the artefact, (which could be a working published tree). The
document's source path moves with it, image_dir_path being reached from
the document's own file rather than from the pod.
Each blob is checked against the sha256 the database recorded beside it.
The two were written together, so a mismatch means the file is damaged.
(the point of having recorded the digest).
A .ssp only describes its images, so those are the ones in the pod it
sits in, and they are checked against the digests it recorded.
That asymmetry arises from of what the two artefacts are: the ocda.db is
self-sufficient and can prove it is looking at the right image, the .ssp
describes a pod it sits alongside.
Found while wiring it: setting the pod directory alone was not enough,
since image_dir_path is derived from the source file's path
(../../image from media/text/<lang>/), so the source path has to be
moved with it.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Process a document from its .ocda.db or .ssp abstraction. A .ssp or a
.ocda.db given as an argument is now a document like any other: read
rather than parsed, and handed to the output writers as the same doc
they already take.
The 35 document sample collection built from its .ssp files is byte
identical to the same collection built from markup: 1457 files, no
differences.
Three things had to be rebuilt rather than read, all of them a value
parsed rather than a structure derived:
- classify_topic_register_arr and its expanded twin. The split is not a
plain one, so the rule moved to sisudoc.ocda.meta.topic_register and
both the yaml reader and this one call it.
- creator_author_arr, which is creator.author split on the ", " it was
joined with.
- title_sub, a copy of title_subtitle made where the header is read, and
what epub3 puts in dc:title id="subtitle".
Fixed ordering bug found by the acceptance test. A heading's own anchor
can be a bare number taken from its text and this can collide with the
ocn of an unrelated object. Whole output comparison found that before
the fix there was one wrong link in one epub's table of contents.
From a .ocda.db, 1446 of 1457 files are identical. The 11 that are not
are the images: a database carries its own image blobs and nothing yet
extracts them, so the five sisu_markup images are not copied and the
epub that embeds them differs. That is the next step and is not a defect
in this one.
--source and --pod2 are refused with a warning rather than half
done, no artefact carrying the markup.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The segment a cross reference to a heading lands on, gixed upstream in
ocda, where the field is created, rather than inferred by the loader.
A heading above level 4 opens no html segment of its own, so a link to
it has to land on the level 4 heading that follows. The build loop
cannot know that when it reads the heading, and says so in a comment of
its own: "for html segname need following lv4 not yet known". It
back-fills tag_assoc when the level 4 heading arrives (lv0to3_tags), and
the answer then lives only in that map.
tags.segment_lv4_is now holds it, resolved in a pass over the finished
head and body sections: walking backwards, the last level 4 heading seen
is the next one for everything above it, which is the same answer the
back-fill gives. Emitted in the .ssp only where it differs from
.segment_html_is, which is every heading at level 4 or below, so it is
sparse: 317 lines over the 35 document reference set, none removed.
Carried in the database as a column of its own, and read back by both
readers.
docHasFromAbstraction reads it instead of working it out. With that, and
on top of the two defect fixes, a document loaded from an artefact and
one parsed from markup agree:
key sets identical, all 35 documents
values 3 entries differ of some 30,000, all of them
_the_title, where the abstraction supplies an epub
segment the parser leaves unset
link targets all 1,651 agree, against 89 differing before any
of this work and 1 after the defect fixes alone
Format stays v1.1. That version is new in this same run of work and
nothing outside spine has read it, so this belongs in it rather than in
a bump of its own.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
reading a .ocda.db was slower than parsing markup source. Taking the
largest markup document sample War and peace, 12,135 objects, optimised
build, before this commit and after:
parse markup 0.87 s 0.87 s
load .ssp 0.047 s 0.047 s
load .ocda.db 1.09 s 0.128 s
The database goes from being 1.25x slower than parsing the document to
6.8x faster.
Two things fixed in the reader were:
First, four queries per object. object_images, object_links,
object_anchors and object_subtoc were queried per object as the objects
were built, each statement compiled fresh from a concatenated string.
For war and peace that is 48,540 statement preparations to collect
138 rows, which is all those four tables hold between them. They are now
four ordered sweeps, kept by object id, so the cost is what the tables
hold rather than what the document holds.
Second, and even more consequentially: d2sqlite3's row["name"] is
indexForName, a linear scan over the statement's columns that calls
sqlite3_column_name and allocates a D string for every column it passes.
At some fifty named reads per object over forty columns that is around a
thousand of those per object, twelve million for the document. The
column name to index map is now resolved once per statement and the
reads are an integer index.
Also here, since it was measured while doing this: sspReadFile no longer
hashes the file it read unless asked (with_digest). Only the database
writer wants that digest and it has the lines already, so every other
read was paying for it.
Existing outputs unaffected.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
.ssp format v0.1 -> v1.1, and the .ocda.db with it: they are two
serialisations of one format and now move together.
The gap this closes, measured against the writers' read set: of the 59
conf_make_meta fields outputs/io_out/ reads, 29 were in neither
artefact. Five are conf.* and stay out, being site and run scoped (urls,
papersize, the search db filename): the same abstraction published to
two sites must take each site's. Of the rest, three are never assigned
anywhere (title_short, publisher, original_publisher; read only by
sqlite.d, always empty) and one is a copy of a field already carried
(title_sub = title_subtitle), so 17 properties actually had to travel
and now do:
make breaks, footer, home_button_text
meta title.edition, date.added_to_site, language.document_char,
original.{title,source,language,language_char},
rights.copyright_{text,translation,illustrations,
photographs,cover,audio,video}
make splits by when it acts, which is worth keeping in mind: these three
are read in outputs/io_out/ and must travel, while italics, bold,
emphasis, substitute and headings are read at parse time and their
effect is already in the objects.
New @source block, and source.* rows in the database, so a reader can
say which markup an abstraction came from rather than working from
stale content in silence:
language the document's own
languages the pod's list, which is what the inter-language links
in html need and neither artefact carried
digest sha256 of the .sst; equals its digests.txt entry
The database adds source.ssp_digest, the sha256 of the .ssp it was built
from, since a file cannot hold its own hash. So the chain
.sst -> .ssp -> .ocda.db is checkable end to end.
The database's metadata table is now filled from the header blocks the
.ssp gives back, not from a second list read off doc_matters. One list,
in ssp.d: a property added there arrives in the database with nothing
else changed, and one that is not in the .ssp cannot be in the database
at all. That was the last place the two could drift; the objects stopped
being able to on 2026-09-07.
Version is checked on load. The major part must match, a newer minor is
accepted (a minor bump only adds properties, and an unknown property
line is ignored). A v0.1 artefact is now refused with a message saying
to regenerate it, rather than loading half populated.
test-abstraction-db.sh now compares the two artefacts' header blocks
property by property, 36 per document on the wealth of networks, where
it previously only checked that a schema.version row existed. Verified
non-vacuous: dropping one property is reported.
Reference regenerated, and the whole diff is this change and nothing
else: 320 lines added, 35 removed over 35 files, being 35 format lines
changed, 35 @source blocks (4 lines each), 35 language.document_char, 35
home_button_text (it has a default), 18 breaks, 16 footer,
4 date.added_to_site, 1 title.edition, 1 original.source.
Existing outputs unaffected.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
| |
ocda (object centric document abstraction)
flag --ocda-db (or --abstraction-db)
replaces --show-abstraction-db
rename results in consequently large diff
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The per document database is now written as <doc_uid>.ocda.db
rather than <doc_uid>.abstraction.db. Shorter, and it says what is
in the file: the object centric document abstraction, not "some
abstraction". .ocda.db pairs with .ssp and cannot be mistaken for
the collection search database (spine.search.db).
test/run-tests.sh runs the four in sequence, one line of result
each, and a summary. It re-runs itself inside nix shell
"nixpkgs#sqlite" if sqlite3 is not on PATH, so this is all that
is needed:
SpinePOD=../../markup/sisudoc-spine-samples/markup/pod-samples/pod \
./test/run-tests.sh ./bin/spine-ldc
The tests are independent (with test-abstraction-ssp.sh run
first):
1 test-abstraction-ssp.sh is first because it is the
one that says whether the abstraction itself moved; if it
fails the others are answering a different question than you
think
2 test-abstraction-ssp-roundtrip.sh reads the committed
reference set, so it is a statement about the current binary
only if 1 passes
3 test-abstraction-db.sh and
4 test-abstraction-db-roundtrip.sh generate both artefacts
themselves and depend on nothing committed
(assisted by Claude-Code)
|
|
|
sisudoc.ocda.abstraction.db_in reads a <doc>.abstraction.db back
into ObjGenericComposite[][string], the same value ssp_in returns
from a .ssp, so a consumer need not know which artefact it was
handed. It mixes in the .ssp reader for that shared document
struct rather than declaring a second one.
--db-round-trip=<file.abstraction.db> reads a database and emits
it as .ssp on stdout, through sspObjectRecord as the other round
trip does. Held against the .ssp written from the same document,
this says whether the two artefacts really carry the same thing:
not a count of fields, as test-abstraction-db.sh does, but the
whole document reconstructed from the database and compared to the
text.
SpinePOD=... ./test/test-abstraction-db-roundtrip.sh ./bin/spine-ldc
PASS: all 35 databases re-emit their document's .ssp exactly
It passed on the first run over the whole sample set, which is evidence
that the database is now field-complete against the .ssp rather than
merely counting the same.
Four tests with different checks:
test-abstraction-ssp.sh the abstraction has not changed
test-abstraction-db.sh the two serialisations agree, field by field
test-abstraction-ssp-roundtrip.sh the .ssp can be read back whole
test-abstraction-db-roundtrip.sh the .db can be read back whole
(assisted by Claude-Code)
|