| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
need a character to split filenames on in certain circumstances.
there problems with use of a colon in filenames, which is legal on posix
but not on Windows."~" fits the bill better being legal on every
filesystem of interest and is unreserved in rfc 3986, needing no escaping in a url.
The reference abstraction is renamed, its content unchanged. Over
the sample collection two filenames move and nothing else does: the
abstraction and database digests are identical, the archive's
members and their sizes are unchanged, and only the member order
shifts, "~" collating after letters where ":" sorted before them. The
document's epub dc:identifier is a v5 uuid derived from the uid, so it
changes for those filenames.
Every document's uid changes in the search database (spine.search.db).
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
dr_document_make in three identifier spellings becomes document_make,
which is what the file has been called since the rename.
The mixin import audit, a template declares its imports at template
scope and they are then visible in every scope it is mixed into (which
is how a template-level split once hijacked a UFCS lookup in code that
had not been touched). Disabling htmlSnippet's imports and rebuilding
names the consumers that were relying on them rather than on their own:
across twelve mixin sites, exactly one. metadata.d now imports the `to`
it uses. The templates keep their imports, their own functions needing
them, and narrowing those is a change of its own rather than a cleanup.
And the standing FIX in source_pod.d: the insert digest line for a non
pod source recorded the insert's path, where every other line in
digests.txt is a filename and a digest naming a file that is not in
the pod under that name cannot be checked against it.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
| |
doc_uid_out ends in ".{lng}", which is right for an artefact written per
language and wrong for one that holds every language of a document.
Rather than strip the suffix at each place that needs the bare name,
build it by the same branches without the language, so the two names
cannot drift apart.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
format 2.0, source.digest covers every markup file, the sha256 over one
line per markup file, "<sha256> <filename>", sorted by filename.
source.digest was the sha256 of the master file as read, before any
insert. This is right for a .sst, which is the whole document. For a
.ssm with inserts .ssi (containing the substantive part of a documents
text) this is close to meaningless.
Considered change of meaning (rather than an addition) and given the
major part of the format version with it: 1.1 becomes 2.0, in the .ssp
header line and in the database's schema.version row together.
A 1.x reader refuses a 2.0 artefact, which is what that check is for.
- Sorted, so the value does not depend on the order a filesystem hands
back a directory.
- By filename (rather than by path), so it does not depend on where the
pod sits, which is the property the pod rebuild comparison rests on:
a pod unzipped elsewhere must still give the same abstraction.
Within one language every markup file lives in one directory, so a
filename identifies it.
Each line is checkable on its own against a digests.txt line or a files
row, which the single value was not. The whole is reproducible with
sha256sum and sort, and was verified that way for a one file document
and for a twenty file one.
An unreadable file is named in the digest rather than skipped. The parse
has failed elsewhere by then, and a digest that quietly left a file out
would claim the document is something it is not.
The reference abstractions are regenerated. Two lines change in each and
no others: the version, and the digest.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Both now take the number from one constant now set at 10,000, the writer
checks it before the archive is written.
Previously the number set for the pod reader was less than 500 members.
The writer had no limit, (so spine could write a pod it would then
refuse to read, reporting too many entries: a good file that looks
corrupt, and only for a document with enough parts).
500 was set when a pod held markup and images. A pod carrying
translation catalogues has a different arithmetic, and the count follows
from how many files a document is made of times how many languages it
has, not from how large the document is:
the_wealth_of_networks 213,405 words, 1 file per language
12 languages -> 49 entries
live-manual 24,724 words, 20 files per language
12 languages -> 506 entries
The big book is not what runs into this; the modular manual is.
live-manual stood two languages from an unreadable artefact.
10,000 covers a hundred-insert manual in thirty languages, about 6,200
entries, with room. It is deliberately well clear of any real document
rather than snug above the largest one known: the two errors are not
comparable, since too low refuses a legitimate document with a message
that reads as corruption, while too high defers to a size cap a moment
later. The count is the weakest of the three guards and is not what
bounds resource use; the per entry and total size caps do that, both
before a byte is written.
Refusing also removes any archive an earlier language left. The writer
runs once per language and only the last pass holds every language, so
it is the last that goes over, and the passes before it wrote smaller
archives that passed. Without this, refusal left a pod missing a
language: an artefact that reads perfectly well and is wrong. A missing
file is an error someone notices.
The arithmetic is recorded beside the constant so the next person can
re-derive the number rather than guess at it.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The pod archive spine writes is now named .sisupod and carries the
same archive inside one zstd frame. The reader has accepted both
since the previous commit, so every pod already published stays
readable and this changes only what is written.
free_culture 2863504 -> 1100180
live-manual 6972418 -> 692942 10.06x
The wrap is at the pod call site and not in createZipFile, which
also writes every epub and every odt.
Best compression from whole-stream (rather than per-member compression).
(for live-manual per-member deflate manages about 3x compared to 10x for
whole pod content compression).
The suffix is named as pod_archive_suffix.
--pod-compression sets the level, 19 by default: a pod is written once
and fetched many times (692942 bytes at 19, 837703 at 9, 2054365 at 1).
(A value that is not a number warns and the default is used).
Output built from a written .sisupod is byte identical to output built
from the markup it came from.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
sisupod using zstd compression recognised by first bytes.
<doc>.sisupod: spine reads one before it writes one, so when the name
changes (from .zip to .ssiupod) every pod already published stays
readable.
A .sisupod is one zstd frame wrapping the archive spine already builds.
The reader unwraps the bytes and hands the same archive to the same
parser, so every guard downstream is untouched: entry names, per-entry
and total size, path depth, escape and symlinks.
Which container it is comes from the first four bytes (rather than the
name). A pod published as a plain .zip reads as it always did, a
.sisupod reads, and either one renamed reads too. The suffix is still
recognised, both spellings, for the argument and for a url.
libzstd is declared rather than bound: provides the whole surface of
fifteen extern C prototypes (there is nothing to generate and no
upstream tree to track, which is the arrangement sqlite3 already has).
dub links it with "libs": [ "zstd" ]; nix needs zstd.out rather than
zstd, whose default output is the binaries and carries no library at
all.
A frame declares its uncompressed size in its own header, and for a pod
fetched over https that number is attacker controlled. The declared size
is checked against a ceiling before a buffer is asked for, a frame that
will not declare one is refused, and what comes out is checked against
what was promised. The ceiling is the extraction limit the archive
reader already applies, so the two bounds agree.
Measured: output built from a .sisupod is byte identical to output built
from the same pod's .zip, 359 files over text, html, epub, odt and .ssp
for three documents, live-manual's ten languages included. A truncated
frame is refused and the document skipped.
free_culture 2863504 -> 1099736 2.60x
the_wealth_of_networks 4294022 -> 1172649 3.66x
live-manual 6972418 -> 675970 10.31x
(assisted by Claude-Code)
|
| |
|
|
|
|
|
| |
Spine now looks for conf/document_make, and bundles that name.
(the dr_ prefix was a sisu-era name to distinguish it)
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Spine names a language as the directory under media/text/ is
named, which for a region is the posix form with an underscore,
pt_BR. A lang, xml:lang or dc:language value is a BCP 47 (RFC
3066) tag and wants a hyphen, pt-BR. Spine put its own code
there unchanged, so a document in a region language produced an
invalid attribute in every one of its xhtml files: 138 epubcheck
errors for a document of ten segments and a navigation file.
src.language_tag is the same language as a tag. The attributes
and the dc metadata now read it. Output paths keep src.language,
since a path is named after the directory, not after the tag.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
bugfix: content image names are now checked in a pass of their own,
before anything is created or written, and one bad name refuses every
image in the artefact rather than skipping the one, which is what the
zip reader does with a zip. The write then re-checks containment.
(prior to this an image name read out of a .ocda.db was written without
checks, and such a database can be fetched over https. A name could
climb with "..", and being handed to chainPath an absolute name dropped
everything before it, so "/x" did not land under the image directory at
all. The zip reader has guarded against this since it was written; the
database reader did not).
carried_names.d holds both rules: a bare filename, as an image is
carried, and a relative path, for the markup and conf a database is to
carry.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A file beginning with a utf-8 byte order mark had the mark read as part
of its first yaml key, so "title:" arrived as "title:" and every
value under that key was dropped silently: the document kept its text
and lost its title. Downstream that showed as an empty dc:title and an
empty <title> element in the epub, which epubcheck reports as two
errors.
The mark is now kept out of the header and body split, and so out of
the parse. It is not stripped when the file is read, because that text
is what source.digest is taken over and the digest names the file as it
sits on disk.
One document in the sample set begins with a mark. It is left as it is:
it is the case this guards against.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A URL ending in .ocda.db is now downloaded and processed in place,
through the path that already did it for .zip: the same curl call, the
same size and timeout limits, the same refusal of local and private
addresses, the same --allow-downloads guard, the same temp file cleanup.
Only the pattern had to widen.
rgx_url_zip ^https?://...[.]zip$
rgx_url_source ^https?://...([.]zip|[.]ocda[.]db)$
downloadZipUrl is no downloadSourceUrl and the download temp directory
is spine-download rather than spine-zip-pod; (the extraction directory
keeps its name, being still only for zips).
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
| |
correct circumstance where regex was incorrectly able to match image
shaped words as well as identified and marked up image
(internal markup).
take all matches.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
| |
Modules and imports rewritten to sisudoc.ocda.* and
sisudoc.outputs.*; dub.json excludedSourceFiles and the
spine:abstraction sub-package sourcePaths collapsed to
./src/sisudoc/ocda.
Verified: nix build .#spine-overlay-ldc clean.
(assisted by Claude-Code)
|
|
|
create new directories under ./src/sisudoc ocda & outputs in order to
separate the document abstraction library from downstream output
processing (stuff broken till paths & modules fixed)
|
