`metric chars|flat|blocks` inside %synth meant the same truth table could be
searched twice for two call sites, and every library block had to spell out
`metric flat` to avoid the default. The cost model belongs to the whole
assembly, so it is one now:
r8forge <file.r8asm> flat (layout free), the default
r8forge --synth-chars <file.r8asm> charge the real .tp walk
flat stays the default because a library block is expanded at dozens of call
sites with different layouts: only a layout free result can share one cache
entry, and it is usually within a few percent (full_add: 95 vs 90 characters).
The chars search is far more expensive (full_add: 415M states, ~21 minutes,
~10 GB), which is exactly why it should be asked for on the command line.
`metric blocks` goes with it: block count is a poor proxy for tape length (and
it was only another way to ask for mode layers).
Implementation: the metric lives in the assembler (ast::synth_metric, default
1) and is set through r8asm::set_synth_metric() before import_file; the CLI in
tests/asm_test.cc parses --synth-chars / --synth-flat. The cache key already
carried metric, so the committed entries still hit: assembling the library and
asm/test.r8asm gives byte identical tapes (28729 characters, byte_op_test all
green, all 13 examples green, synth_test still 8/8).
A `metric` line now fails with a message that names the flag instead of being
silently ignored. examples/full_add_flat.r8asm and full_add_flat_spaced.r8asm
are removed: they only existed to demonstrate the per block metric (the flat
default makes them copies of full_add.r8asm).
Also repairs the mojibake in README's metric section: an earlier tool wrote
\x0c / \x08 / \x0d where \f, \b and \r belonged, which ate characters out of
"flat", "blocks", "biz/flb/rnz" and "rnz". The section is rewritten for the
flag, and the %synth numbers in README/SPEC/AGENTS now match what the tree
prints.
New cache entry c16bf083bb0ccbc8.txt is full_add's stat table with mode layers
+ flat, which full_add_stat.r8asm now produces.
clb_digits needed to name a different cell every round, which %rep cannot do: it
copies identical text N times and has no loop variable, and there is no %if
either, so the three shift steps were spelled out by hand and the macro was
only correct for 11 bit cells.
%for k 3 ; k is 0, 1, 2 inside the body
step k
%endfor
%for <name> <count> binds <name> to 0..count-1 as a compile time constant that
any expression can use: %rep counts, .tp targets, macro arguments. The loop
scope is a copy of the current scope plus the name, so a %for inside a macro
body still sees that macro's %1, %2, ... %rep and %for nest in each other
(both closers count as a closer). %rep itself is untouched.
With it, clb_digits lost its hardcoded chain: the digit count, the shift count
and now the constant builds (tails written as _CELL_SIZE - 5 / _CELL_SIZE - 3)
all follow _CELL_SIZE. Checked at 16 bits with a five cell region: 4660 gave
'0' '4' '6' '6' '0' and 7 gave '0' '0' '0' '0' '7', so the count really did
become five. 65535 hangs in clb_divmod (6553 rounds, over the 5M step budget)
-- that is the divider's O(N/D), not digits. Back at 11 bits everything is
unchanged: all eleven values, hash 28729, byte_op and docs green.
SPEC 3.5 documents the directive; it is a local extension like %synth.
The cell width is cell_size everywhere else now -- cell_size in the AST, in
ROT-8-viz's rot8_config, and the docs -- but the default in geometry.hh was
still kDefaultCellBits, and the four VMs spelled it that way too.
* include/r8forge/geometry.hh: kDefaultCellBits -> kDefaultCellSize.
kDefaultCellMask keeps its name: it is a mask, not a size.
* src/derive.cc, tests/asm_test.cc, tests/synth_test.cc, tests/tape_run.cc,
tests/byte_op_test.cc: same rename, and the local shorthand kBits in
byte_op_test.cc is now kCellSize so the test file does not put the old
word back.
* AGENTS.md and README.md follow. The cross check bullet now describes the
workspace ROT-8-viz as it actually is (fork dsh/ROT-8-viz, branch main,
runtime --cell / --mem over cell_size / mem_size), instead of the feature
branch it used to point at, which no longer exists.
No behaviour change. All six self checks re-run green: 28729 character hash
tape, synth_test 8/8, both verify_ tapes, byte_op_test (clb_mul 3977 chars,
65536/65536, "byte level library ok") and the macro docs test. The hash tape
still gives 1322 1786 1600 5 94 127 from r8forge_derive, r8forge_tape_run and
ROT-8-viz's rot8_test -- three implementations, one answer.
The cell width was CELL_SIZE on the C++ side, and the searcher's CellBit count
was CELL_ARGC. Both now match what the code and the docs around them already
called them:
* include/r8forge/r8asm.hh + src/r8asm.cc: CELL_SIZE -> cell_size
(set_cell_size assigns this->cell_size, its parameter already had that
name);
* include/r8forge/r8veritas.hh: the template parameter CELL_ARGC ->
cell_argc, 34 uses, so the searcher reads r8veritas<cell_argc>;
* AGENTS.md says which name means what, and why the name .r8asm programs
see is still _CELL_SIZE: that one belongs to the macro library, not to
the C++ side, and changing it would rewrite every macro in asm/;
* the cross check entry now says which ROT-8-viz is which: the author's fork
with --cell/--mem, and the upstream starcells/ROT-8 tree in the workspace,
whose geometry is compile time cell_size/mem_size -- it is 11 / 259 too
now, and `rot8_test <tape>` against `r8forge_derive ... -1` still gives
1322 1786 1600 5 94 127 on the hash tape.
All six self checks pass unchanged: 28729 character hash tape, synth_test 8/8,
both verify_ tapes, byte_op_test 65536/65536 for clb_mul and the rest of the
library, macro docs test green.
The assembler already took the width as an argument (import_file(path, cell_size))
and exposed _CELL_SIZE / _CELL_MAX / _ROL_STEP to programs, so every macro that
writes %rep _CELL_SIZE follows the width on its own. What was missing was a
single place saying what the default is, and VMs that agree with it.
include/r8forge/geometry.hh now holds kDefaultCellBits = 11,
kDefaultMemoryCells = 259 and kDefaultCellMask, and the four VMs use it:
* cells are uint16_t, the tape is kDefaultMemoryCells long
* ROR is (v >> 1) | ((v & 1) << (kDefaultCellBits - 1)), not the 8 bit version
* input and output mask with kDefaultCellMask
The tests now state their expectations in terms of the width instead of a byte
(kCellMask, kBits, kCellMod). Two things were genuinely wrong for 11 bits and
are fixed: clb_neg_* and clb_inc_* were compared against 256 - a and 0xff, and
asm/test.r8asm set B with .xor (~126)+1 -- .xor takes its constant modulo the
cell width, so that is 1922 in an 11 bit cell, not 130.
Results at 11 x 259:
asm/test.r8asm 7262 characters, out 127 130 0 257 0 1088 19
(127+130 = 257, and 200*200 = 19*2048 + 1088)
r8forge_byte_op_test all 17 operations pass
r8forge_synth_test all eight combinations ok (bit level is width free)
r8forge_macro_docs_test 41 macros in 17 files
asm/fibonacci.r8asm 11959 characters, 13 terms now (no wrap at 256)
ROT-8-viz moved its defaults to 11 x 259 in the same change, so the tapes cross
check: both implementations print 127 130 0 257 0 1088 19 for asm/test.r8asm.
Tapes grow when the width does (raw_add 1647 -> 2199 characters), because the
%rep _CELL_SIZE loops are longer.
AGENTS.md gains a section on how the width flows through the tools, including
the trap that 2048 % 256 == 0 makes several 8 bit expectations still pass by
accident.
r8asm::ast::CELL_SIZE machine word width in bits, run time value
(import_file's second argument, 8 in the CLI)
r8veritas<CELL_ARGC> how many parameter cells a synth block has,
compile time, selected by 'params N'
set_cell_bits went back to set_cell_size for the same reason, and the .r8asm
visible names (_CELL_SIZE, _CELL_MAX, _ROL_STEP) are unchanged, so lib.r8asm
keeps working as written.
This time the string literal used by %rep and by the .xor masking is the one
the C++ member is now called too, which removes the trap the previous rename
walked into.
Regressions after the rename:
asm/test.r8asm exit 0, 1666 characters, 7/7 cached
examples/full_add_layers.r8asm eight combinations ok
derive bit_raw_add same table as the hand written version
clang-cl build of all five targets, and the clang built binaries produce the
same 1666 character tape and the same table.
There were two unrelated things under one name:
r8asm::ast::CELL_BITS the machine word width in bits, a run time number
(import_file(path, 8)), used by %rep and the .xor
argument masking
r8veritas<PARAMS> how many parameter cells a synth block has, a
compile time template argument selected by 'params N'
The first is now CELL_BITS, the second PARAMS. What a .r8asm program sees does
not change: _CELL_SIZE / _CELL_MAX / _ROL_STEP keep their names, because
lib.r8asm uses them.
Answering the actual question along the way: the machine word width is a run
time parameter (import_file's second argument, hardcoded to 8 by the CLI), not a
template argument -- the comment in r8asm.hh even apologises for that. The
synth block's 'params N' is the other one, and only that goes through the
template switch (1..8).
The first attempt at the rename also rewrote the _CELL_SIZE string used by %rep
into _CELL_BITS, so the lookup threw and the assembler died with a fastfail on
any file containing .xor or %rep. Caught by regressing test.r8asm (exit
0xC0000409, no tape); the string is restored and asm/test.r8asm is back to 1666
characters with all seven blocks cached.
%synth could optimise the real tape characters, but that price depends on where
the cells sit, so the answer was only good for one layout and the cache could
only be shared between call sites with the same addresses.
metric chars tape characters including the .tp walk layout dependent
metric flat characters without the walk layout independent
metric blocks block count layout independent
'flat' is the step-free mode: it counts biz/flb/rnz and the ops, ignores how
far apart the cells are, so one search result and one cache entry serve every
layout. The price is a slightly longer tape when the layout happens to suit a
different program: 90 characters for chars, 95 for flat on the full adder.
Checked: examples/full_add_flat.r8asm hits the cache on the second run, and the
same block with 'layout 0 2 4 6 8 10' added hits the *same* entry and emits a
byte-identical tape (the key drops the layout for the layout-free metrics).
Both pass the eight combination check.
Found with scoop's llvm 23.1.5 (clang-cl), which is stricter than MSVC about
non-type template arguments:
candidate template ignored: substitution failure: deduced non-type
template argument does not have the same type as the corresponding
template parameter ('unsigned long long' vs 'int')
block_cost deduced its N from std::array<int, N>, and that deduces size_t;
clang refuses to convert size_t to the int parameter, MSVC allows it. N is now
size_t. Every target (r8forge, synth_test, derive, cost_par, cost_search)
builds clean with clang-cl and the clang-built binaries assemble test.r8asm to
the same 1674 characters, pass the eight combination check and derive the same
bit_raw_add table.
Two other portability fixes in the same pass:
* cost_search used _strtoui64, which is MSVC only -- now std::strtoull
* the CMake project now finds Threads and links Threads::Threads, which
clang and MinGW need to pull in the thread runtime
* if spawning a worker throws (no thread runtime available) the search now
carries on with the threads it did get instead of failing
The sequential search is memory bound on its hash table, so the split that
works is inside a Dial bucket, not across the graph:
* the table is read only for the whole bucket, so threads probe it with no
locking -- 99.7% of the work, since only about one edge in four hundred
lands on an unseen state
* misses and dist improvements are buffered per thread and applied by one
serial pass after the bucket
Cost order is untouched, so the answer is identical. Measured at limit 60:
threads wall states
1 12.40s 7,671,626
8 2.68s 7,671,626
16 2.20s 7,671,626
32 2.14s 7,671,626
and at the full limit 82 (the run that used to take 1350s): 162s, 414,631,673
states, cost 75 in 6 blocks -- the same program.
%synth now uses this by default. examples/full_add.r8asm assembles in 166s
instead of ~23 minutes and still produces the 90 character tape, checked on the
machine for all eight combinations.
The earlier attempt -- one worker per possible first block -- stays in
r8veritas.hh behind opt.threads but off by default; it duplicated the middle of
the graph and cost three to four times the work.
Tried the obvious split -- one worker per possible first block, since a
program's first block is unique and the answers cannot overlap -- and measured
it on the full adder at limit 82:
workers wall total CPU peak memory
1 1350s 1350s 11 GB
8 ~10min 4600s 14 GB
32 -- 4870s 17 GB
So the answers do not overlap but the *search* does: every prefix walks into the
same middle of the graph, and with a table per worker that middle is explored
once per worker instead of once overall. Three to four times the work and more
memory, for a modest wall clock win.
Kept the code (opt.threads > 1) because it is correct and it is the right shape
for a machine with more memory, but the default is one worker. The parallel
split that should actually pay off is inside one shared table: expand the
current Dial bucket across threads, insert through a sharded table. That is
the next attempt, not this one.
A synth block used to spell out which parameter is an input and which is an
output. That was redundant: with map, 2^K entries already mean %1..%K are the
inputs and the value's bits land on the parameters after them; with a raw
table, nothing needs saying at all. So 'inputs' and 'outputs' now fail with a
message pointing at map/stat, and the map form derives both.
Added 'stat' as the fully explicit form: 2^params values, one per input
combination, which is exactly the state table the search consumes.
examples/full_add_stat.r8asm is the same adder written that way and produces a
byte-identical tape to the map version (95 characters), both pass the eight
combination check.
%synth used to always run both searches, so every assembly of the full adder
sat there for twenty minutes. A block can now say 'mode layers' to stop after
the layer search (7.7 seconds) or 'mode tape' (the default) to pay for the
shortest tape. examples/full_add_layers.r8asm is the fast one, 95 characters
and still checked on the machine.
The lexer also no longer chokes on a UTF-8 BOM. Every Windows editor writes
one, and the first token came out as a garbage prefix, which crashed the
assembler with an access violation rather than saying anything useful.
README grew a getting-started walkthrough: build, a complete adder.r8asm, what
the two stderr lines mean, how to drop a synth block into a library, and the
three things that bite (inputs first, don't call them in/out, map length).
The layer search still runs first (7.7 seconds) but only to get a bound; the
cost search then takes over and its answer is what goes on the tape. A caller
that wants its own layout can say 'layout 0 2 4 6 8 10' inside the block.
examples/full_add.r8asm now assembles to 90 characters of tape instead of 95:
cost 75 in 6 blocks after 415M states and about 21 minutes, against the 82 the
five layer program needs. Assembly getting slower is the whole point -- it is
the price of the shortest tape.
For scale: one bit_raw_add slice is 179 characters with the cells adjacent, and
the synthesised slice is 82. A whole raw_add call is 1650 characters today, so
eight synthesised slices would land near 700.
First version scanned ahead for the next nonempty bucket but stopped one max
edge weight out, so a sparse frontier ended the search early and claimed
"nothing under the limit" while the queue still had work. Scanning further is
also wrong: the ring reuses a slot every ring.size steps, so a far bucket sits
where a near one would be. Plain Dial now: step one cost unit at a time.
The growth table in the README was measured with the broken version and was
wildly optimistic. Real numbers from the fixed build, full adder with
adjacent cells: cost 58 -> 80M states, 66 -> 220M, 70 -> 280M, about 27 bytes
per state and 650k states/s, so cost 75 / 450M states / 12GB is roughly the
ceiling on this machine.
Validated solve_cost on the small problems where the answer is known by hand:
copy comes out at cost 17 (sacrifice 0 into {2}, then sacrifice 2 into {0,1}),
which is the two block program, and it replays ok.
Layers are not characters. solve_cost runs Dijkstra over the same state graph
with an edge weight of what the block actually assembles to:
w = 3 (biz + flb + rnz) + 2 * span(cell addresses) + sum(op)
op = 1 for flb, 4 for set_1
Dial buckets, pop in cost order, the first pop of the goal is the shortest
tape. Give it max_cost and it becomes branch and bound: anything past the
bound never enters the queue.
Exhausted regions on the full adder with adjacent cells:
max_cost 20 1,086 states 0.0s
max_cost 40 128,528 states 0.1s
max_cost 50 999,448 states 0.7s
max_cost 60 7,671,626 states 12.8s
max_cost 65 20,153,459 states 47.6s
About 2.6x states per 5 cost, about 32 bytes per state, so max_cost 80 is
~350M states / 11GB and 82 would be ~600M / 19GB. The block weight leaves out
the hop from the previous sacrifice (that would make the cost depend on
history and Dijkstra would stop working); the emitter adds it when it reports
the real length -- the five layer program is 82 in the model, 87 on the tape.
The first version of the bucket ring only looked one max-edge-weight ahead for
work, so it declared 'nothing under the limit' while the queue still had
states further out. That made the small limits look sound when they were not;
advancing to the next nonempty bucket fixed it.
The searcher stops being a separate program: %synth is an assembly-time
directive. It describes a truth table, the assembler runs the search while
parsing, and the block list comes back as ready-to-paste r8asm.
Merging is what makes the cost model honest. The price of a layer is
cost = 2 (biz + rnz) + sum(move distance + op cost)
and the move distance depends on the call site's cell layout, not on anything
the assembler knows in advance: the same five-layer adder is 87 characters
with the cells adjacent and 136 with them two apart. Only the side that sees
the layout can price it, which is now the assembler itself.
Working: the assembler (asm/test.r8asm still assembles to the same 1714
characters), the searcher (bit planes + backward one-step lookup, full_add in
5 layers, proven shortest in 7.2s), and %synth parsing plus the search itself.
Not working yet: splicing the synthesised body into the tape -- a synth call
fails loudly with 'unknown macro' instead of emitting anything, so nothing can
silently assemble wrong. The two ways to finish it are in the README.
Also carries over the CMake fix: no set(CMAKE_CXX_FLAGS ...), which used to
drop /EHsc, push -O3 into Debug and fight /RTC1.