Sync Packages Across Dark Instances

[StachuDotNet]

Ready for preliminary review, but not yet full -- I'm still tightening PR description, testing across machines, tightening code changes, etc. Close.

Summary

This adds package sync to Dark: run more than one instance — two of your own machines, or you and a
collaborator — connect them, and package edits, branch structure, and conflict resolutions flow between them over
HTTP, converging to the same state on every instance. It's opt-in and, once set up, automatic; local and offline
stay the default.

I built it on the storage we already have rather than adding a new mechanism. Darklang already stores packages as
an op log (package_ops), with every queryable table — functions, types, values, locations — folded from that
log as a projection. So sync is just: a peer serves its new ops, you append them, and they fold in exactly like
your own edits. No merge engine, no separate wire format — the op log is what travels.

Making that deterministic and safe needed a few supporting pieces, also here:

• an authoring timestamp (origin_ts) on each op, so concurrent edits resolve last-writer-wins the same way on
  every instance;
• content-addressed op ids that survive an op-format change (meaning-stable hashing), so content-addressing keeps
  working across versions;
• a synced log of human conflict-resolutions, so an override propagates;
• native builtins that read + serve the logs at speed;
• a Release version coordinate, so peers refuse to sync across incompatible versions.

The F# core stays small; the transport, peer config, daemon, and CLI are all .dark.

A few decisions I'd flag for review: connecting a peer is a full trust boundary for now (peers fully trusted —
see TODOs); the origin_ts MIN-reconcile on receive is what makes LWW converge (worth a careful look); and I kept
the three logs as plain SQLite tables rather than a value type (reasoning under Internals). I've tried to be
honest below about what's a v1 tradeoff vs. actually done.

Supersedes the earlier drafts #5676 and #5684. Two things changed deliberately since those: sync is
wholesale, not per-branch — #5684's follow/private controls are gone (the private promise leaked
through the content-addressed blob channel, so rather than ship a broken guarantee, every branch syncs and
proper scoped sync is a future PR); and transport is HTTP-only — #5676's file mode (dark sync pull <peer.db>) is dropped (a peer is a running instance, not a .db file). This branch is also rebased onto
current main and organized as five reviewable commits (store · builtins · Dark surface · tests · one
unrelated dark view fix — see below).

At a glance

• Sync layer — new, mostly .dark. dark sync connects peers and pulls their new ops over HTTP; dark sync serve serves yours; dark sync daemon runs both on a poll interval. Content-addressed ops make a re-pull a
  no-op; origin_ts last-writer-wins makes concurrent edits converge regardless of arrival order.
• The op log, adjusted for sync. package_ops/branch_ops already existed. Each op now carries an
  origin_ts (its author's stamp, preserved across sync) for LWW convergence; package_ops's primary key
  becomes (id, branch_id) (fixing a ghost-function bug where the same op on two branches collided).
  Projections are unchanged — still folded from the log, still regenerable.
• resolutions — a new synced log. Two instances editing the same name auto-resolve by LWW, but the
  divergence is recorded, never silent; overriding it mints a Resolution that syncs, so every peer converges
  on your pick.
• Wholesale, consistent branch sync. Every branch syncs to every instance you connect — a branch whatever
  is the same whatever (same id + name) everywhere, never scoped by peer or account. Per-branch controls
  (follow/private) were dropped as a half-measure; scoped sync is a proper future PR.
• Meaning-stable hashing. An op id hashes a function's meaning, not its bytes: let/lambda/match binder
  names are alpha-normalized before hashing, so a re-serialize in a new op format keeps ids stable. This is how
  computeFnHash/computeValueHash work — not a separate step.
• Native op builtins. packageOps/branchOps/resolutions {Read,Append}Native assemble records in F#
  (like Traces), so serving thousands of ops is milliseconds, not per-row interpreter overhead.
• Release version coordinate. A version stamp in the store; a peer on a different release is refused with a
  plain message, and the migrator carries the log forward and re-folds projections in the new shape.
• Synced values self-heal. A value arrives from a peer unevaluated (rt_dval is NULL until it's run); the
  store now evaluates any such values on the next startup, and the value read is null-safe — so a not-yet-
  evaluated synced value returns cleanly instead of crashing a command with an internal NULL error.
• Perf, bundled (general wins, surfaced by sync). listAppend O(n²)→O(n) — every large List.map in Dark
  was quadratic; native log reads took a 500-op read from 57s to ~1s.
• Tested across real instances. A fresh-instance harness spins up genuinely separate stores in a couple of
  lines; scenarios cover convergence, cross-store conflicts + resolution, bidirectional pull, branch
  sync/merge/rebase-convergence, branch-identity consistency, and pagination. Full suite green.

---

UX and design

Quick start

Two machines on a tailnet, syncing in under a minute — author a value on one, watch it appear on the other:

# feriel is reachable:
feriel$ dark sync serve                       # serving on :9000 (foreground)

# you connect to feriel, then turn on automatic sync:
you$ dark sync connect http://feriel.tailnet:9000
you$ dark sync daemon start                  # first run: a short settings dialog, then pull+serve in background

# feriel authors and commits:
feriel$ dark val Feriel.Demo.greeting = 99
feriel$ dark commit "add Demo.greeting"

# …within the poll interval it lands on yours, attributed to Feriel — no manual step:
you$ dark view Feriel.Demo.greeting           # → val greeting = 99

Nothing crosses the wire but ops — no .db files, no ATTACH.

The whole surface (dark sync with no args pulls now; dark sync help prints this):

dark sync — keep your packages in step with your other instances, over HTTP.
Local-first: sync is opt-in, and once you connect a peer it stays current.

  dark sync                    pull now from every instance you're connected to
  dark sync connect <url>      start syncing with an instance (e.g. http://feriel.tailnet:9000)
  dark sync disconnect <url>   stop syncing with an instance
  dark sync serve              let peers pull from you (add --port N; default 9000)
  dark sync status             who you're syncing with, and how current each is
  dark sync pull <url>         pull once from a single instance

All your branches sync wholesale — a branch is the same on every instance you connect.

Automatic (background, and across reboots):
  dark sync daemon start       pull from + serve to your peers on an interval
  dark sync daemon status      whether it's running, and with what settings
  dark sync daemon config      view/adjust interval, port, mode, on-boot
  dark sync daemon stop        turn automatic sync off

When two instances edit the same name, sync auto-resolves (latest wins) but records it:
  dark conflicts               review + override sync divergences

How it works

   Instance A                       the wire (HTTP, JSON)                    Instance B
 +-------------+   GET /sync/events?package=&branch=&resolution=   +-------------+
 |   op log    | ------------------------------------------------> |   op log    |
 | packageOps  |     one bounded batch per log, since a cursor     | append,     |
 | branchOps   |                                                   | then FOLD   |
 | resolutions | <------------------------------------------------ | into the    |
 |  -> FOLD    |          (B serves its own new ops back)          | projections |
 | projections |                                                   | = CONVERGED |
 +-------------+                                                   +-------------+

Sync is: read a peer's new events since where you left off, append them to your own log, where they fold in
exactly like your own edits. A peer serves its committed events since a cursor (as JSON); you GET them and append
them, which inserts each op and folds it — in F#, the same playback a local edit uses.

Two properties make this converge:

• Idempotent. Ops are content-addressed, so a re-pull inserts nothing new and folds nothing.
• Order-independent. The fold resolves each name by origin_ts (last-writer-wins — a bind that predates the
  current one is skipped), so concurrent edits reach the same state on every instance regardless of arrival order.

The subtle bit — the one I'd most want reviewed — is that timestamp. A received op carries its author's
origin_ts (a fresh local stamp would be wrong for replication). Because an op id hashes only content, two
instances that make the same edit stamp it independently — so on receive I reconcile the stamp to the earliest
(MIN) across instances, giving every store one canonical origin_ts per op. That's what makes LWW pick the same
winner everywhere.

Connecting, pulling, serving

The CLI reads like Dark — plain, honest, no jargon. Connect a peer (a .db path is refused — a peer is a running
instance), then pull:

$ dark sync connect http://feriel.tailnet:9000
Now syncing with http://feriel.tailnet:9000 — reachable (10843 ops)   # probes /sync/health up front
$ dark sync
Pulled 3 changes from 1 instance
$ dark sync
Already up to date
$ dark sync
Couldn't reach 1 of 2 instances       # an offline peer never masquerades as up-to-date

connect probes the peer immediately, so a dead port or typo'd URL is caught right there (Added <url> — can't reach it yet…) instead of surfacing later on a silent pull.

N in "Pulled N changes" is what actually applied — a re-pull, or a first pull of a shared base, reports 0, not
the wire size. To be reachable, dark sync serve prints the address peers actually use:

$ dark sync serve
Serving sync on http://localhost:9000 — peers can pull from you here.
  Remote peers use your tailnet address, e.g. http://you.tailnet:9000
  Keep it running (serve on boot):  dark sync daemon start
  Press Ctrl+C to stop.

Peers are URLs only; cursors live locally (sync_cursors_v1) and never sync, so an interrupted pull resumes where
it stopped.

Branches — wholesale + globally consistent

Branch structure is just another synced log: a peer's branchOps (create/commit/merge/rebase) travel before its
package ops, so an incoming package op's branch_id always resolves on the receiver. Every branch syncs
wholesale — a branch whatever is the same whatever (same id + name) on every instance you connect, never
scoped by which peer or account it came from, and you never have to think about it. dark sync status shows who
you sync with and the branches that travel:

$ dark sync status
Syncing with 2 instances (run `dark sync` to pull the latest):
  http://feriel.tailnet:9000    (pulled)
  http://reef.tailnet:9000     (not pulled yet)
  3 branches: main, feature-parse, experiment

Per-branch controls (follow/private) were removed. They were a half-measure — the server-side private
control didn't actually hold (a private branch's content still leaked through the content-addressed blob
channel), so rather than ship a broken privacy promise, sync is wholesale now and proper scoped sync is a future
PR.

Concurrent rebases of the same branch converge deterministically (LWW on a synced origin_ts), so branch base
state agrees on every instance regardless of arrival order.

Conflicts + resolutions

Two instances can bind the same name to different content at once. LWW picks a deterministic winner — but
silently dropping the other edit isn't acceptable, so as received ops fold, every such divergence is recorded
in sync_conflicts (a local review log, never synced) with both candidates and the winner. Auto-resolution still
happens; it's just never silent:

$ dark conflicts
1 unreviewed conflict:
  Feriel.Util.parse   active: 9f3a1c2b (theirs, feriel)   yours: 4d8e1f0a · theirs: 9f3a1c2b

$ dark conflicts show Feriel.Util.parse
Feriel.Util.parse — two versions (fn):

  yours   4d8e1f0a
      let parse (s: String): Int = Stdlib.Int.parse s |> unwrapOr 0
  theirs  9f3a1c2b  (active — auto:last-writer-wins)
      let parse (s: String): Int = Stdlib.String.trim s |> Stdlib.Int.parse |> unwrapOr -1

$ dark conflicts keep-mine Feriel.Util.parse
Kept yours (4d8e1f0a) at Feriel.Util.parse. This resolution will sync to your peers.

show renders each side's actual source (not just its hash), so keep-mine vs keep-theirs is an informed
decision rather than a coin flip. ok accepts the auto choice; keep-mine/keep-theirs override it. An override mints a Resolution — a thin
overlay binding a contested location to a chosen reference, with a fresh at stamp so it competes in the same
timestamp-LWW that orders bindings. It is not a new op (re-emitting the SetName would collide on content hash).
The effective binding is fold(package_ops) under LWW, then resolutions applied last. Resolutions sync as the
third log, so your "keep mine" decision propagates and every peer converges on your pick.

Automatic sync — the daemon

dark sync daemon start | stop | status | config is automatic sync as one toggle over both the pull daemon and
the serve daemon. The first start runs a short interactive dialog; later starts are a one-line note. It's
EOF-safe: piped or no-TTY falls through to defaults without blocking.

$ dark sync daemon start
Setting up automatic sync. Defaults:
  · pull from peers + serve to them, every 30s
  · serve on port 9000 — all branches, open to anyone who can reach you
  · start automatically on boot
Adjust these settings? [y/N]

$ dark sync daemon status
Automatic sync:
  pull    sync: running (pid 4821)
  serve   sync-server: running (pid 4822) :9000
  mode: pull+serve · every 30s · port 9000 · on boot · settings: dark sync daemon config

Mode is pull+serve / pull-only / serve-only; settings live in cli-config and are adjustable any time with
dark sync daemon config <key> <value>. (dark version shows the Release coordinate, so you know whether you're
release-compatible with a peer.)

---

Internals

The F# core stays small (PT/RT plus the hot loops); most of the surface is .dark.

The op logs — plain SQLite tables, read/written by native builtins

package_ops and branch_ops already existed as Darklang's package storage. This PR keeps them plain tables (no
bespoke "event log" type) and adds resolutions as a third:

• package_ops — authored package content (add/rename/deprecate fn·type·value). Adjusted: primary key is now
  (id, branch_id) (an identical op on two branches used to collide on a bare id and vanish — a ghost-function
  bug), and each row carries origin_ts.
• branch_ops — branch structure, applied before package ops so branch_ids resolve. Adjusted: carries
  origin_ts (rebase LWW). (Branch identity travels inside the op blob, so no side columns are needed.)
• resolutions — new: synced human/policy overrides, applied last so an override wins.

Each is read as a Stream<Event> after an opaque Cursor, built natively — the F# builtin assembles the records
directly (like Traces), so serving thousands of ops is milliseconds. Folding is invisible to non-F# code: ops
just play back when appended.

There's deliberately no value type wrapping these tables. I wrapped them in a general EventLog value type
at first — mirrored across every serializer, typechecker, and interpreter — but a perf spike settled it:
Stdlib.Sqlite already reads the tables into Dark at roughly native speed, so the eventual path to a Dark-managed
store is "plain tables + Stdlib.Sqlite," and the native fold is a swappable optimization over SQL. The value
type earned neither, so I removed it — the logs are plainly tables. (See TODOs for the eventual Dark-side fold.)

Convergence: one LWW rule, one canonical timestamp

Lww.isStale is the single timestamp-LWW staleness predicate that the fold, conflict detection, and the resolution
overlay all share, so the convergence rule can't drift between them. On receive, receiveOps reconciles each op's
origin_ts to the MIN across instances (so independently-authored duplicates converge), folds, then
detectDivergences records same-name conflicts — firing only on ops about to fold, so a re-pull raises no phantom
conflicts.

Meaning-stable hashing

An op id is a content hash of a canonical serialized form, and that form must not depend on incidental
bound-variable names — two functions identical up to a let/lambda/match binder rename are the same function and
must hash the same. So computeFnHash/computeValueHash rename every binder to a canonical $0,$1,… (in a
fixed structural traversal) before serializing. This lives inside the hashing module — it is how a fn/value
hashes, not a separate pass. (Parameters are already positional EArg index, not hashed by name.) The payoff:
an op-format re-serialize keeps ids stable, so content-addressing — and therefore sync's idempotency — survives
format changes.

Changes by layer

• LibExecution — nearly untouched; PackageRefs.fs gains references to the sync record types (Cursor /
  Event / Commit / …) that live in stdlib/eventLog.dark.
• LibDB — Seed.fs: receiveOps (append, MIN-reconcile origin_ts, fold, detectDivergences),
  eventsSince / branchOpsSince (native reads), receiveBranchOps. Lww.fs: the shared staleness predicate.
  Conflicts.fs: detectDivergences, the sync_conflicts local review log, record/acknowledge/currentBinding.
  Resolutions.fs: the synced overlay (mk / recordAndApply / resolutionsSince / receiveResolutions),
  domain-typed (location: PackageLocation, choice: Reference). BranchOpPlayback.fs: rebase LWW-gate.
  New tables: sync_conflicts (local), resolutions (synced).
• Builtins — Libs/Sqlite.fs: the native read/append builtins for the three logs, plus the raw Stdlib.Sqlite
  floor. Libs/Conflicts.fs: behind dark conflicts. Bundled perf: Libs/List.fs listAppend O(n²)→O(n) —
  two already-valid DLists concat validly iff their ValueTypes merge (O(1)), so map/filter/reverse stop
  being quadratic.
• Dark — stdlib/sqlite.dark: the raw floor (exec/query + injection-safe …P variants, typed Value)
  plus a thin convenience layer (column/queryOne(P)/scalarInt(P)/textField/intField) so
  callers ask for a column/scalar/field. stdlib/eventLog.dark: the typed read/append API over the three tables.
  sync.dark: wireSince, pull, per-(peer,kind) cursors, the blob channel, the poll daemon;
  sync/server.dark: /sync/health + /sync/events. cli/sync.dark: the whole dark sync surface;
  cli/conflicts.dark: dark conflicts; cli/ops.dark: dark ops (inspect the op log live from SQLite).

One unrelated fix (isolated in its own commit)

Rebasing onto current main surfaced a pre-existing regression there: cli/utils/syntaxHighlighting.dark calls
Stdlib.List.collect, which doesn't exist — so dark view errors on plain main with "List.collect not
found". This PR's last commit replaces each |> List.collect f with |> List.map f |> List.flatten (same
semantics) to unbreak view. It's kept as a separate, clearly-labeled commit since it's not sync — happy to
split it into its own PR if you'd rather land it independently.

Tests

Rebased onto current main, the full suite with DARK_E2E=1 is green in one run — 9,956 passed / 0 failed /
0 errored (the self-heal above is what lets the real-HTTP E2E tier seed from an unevaluated store without
tripping).

• MultiInstance.Tests.fs — a fresh-instance harness (freshInstance / seededInstance, a fast copy of a
  seeded baseline) + a test-only swappable connection (LibDB.Sqlite.useStoreForTesting) that repoints all of
  LibDB at an instance. TRUE multi-store scenarios: branch sync, branch create+merge, package convergence,
  cross-store conflict + LWW + recorded divergence, bidirectional pull, pagination, resolution convergence,
  rebase convergence (order-independent), branch-identity consistency, deprecation convergence, and a
  resolution surviving a projection rebuild. It also carries an in-process CLI
  driver (runCli, the same executeCliCommand seam CliTraces.Tests.fs uses) so a test can build a real
  store state and assert on what the CLI actually renders — including a regression for the dark conflicts
  display crash (a genuine recorded conflict must render, not throw).
• SyncScenarios.Tests.fs — fast one-store scenarios (order-independence = convergence).
• OpsProjections.Tests.fs — receiveOps idempotency/convergence, the origin_ts MIN-reconcile, detection,
  the full conflict → resolution flow.
• SyncE2E.Tests.fs — a flag-gated heavy tier (OFF by default; DARK_E2E=1 ./scripts/run-backend-tests --filter-test-list "SyncE2E"). Spins up real dark processes, each its own rundir + data.db, and syncs
  over real HTTP — the layer the in-process tests can't reach (/sync/events server, the real pull loop +
  cursors, process isolation). Scenarios: basic sync + convergence + idempotent re-pull, the conflict race →
  keep-mine → converge, and offline-peer honesty. When the flag is absent it's a single skipped test, so a normal
  run neither builds the exe nor pays the cost.
• Testfiles — cli/sync-per-branch.dark (the blob channel + adaptive poll interval), stdlib/eventlog.dark,
  stdlib/sqlite.dark.
• Manual two-instance validation — beyond the suite, I drove the whole thing by hand: two logged-in accounts,
  separate stores, real HTTP, cold first-sync with no cursor priming. Cold value sync → converge → usable;
  concurrent-edit conflict → dark conflicts → keep-mine → converge; the background daemon auto-pulling a peer's
  new value within seconds; a branch keeping the same uuid + name on both instances; a function authored on one
  instance pulled and executed on the other; and repeated re-pulls staying idempotent (op count + hashes
  stable). The core "two people author and sync" loop works end to end.

---

TODOs (not in this PR)

• Trust model. Connecting a peer is a full trust boundary for now: no op-integrity check, synced values
  auto-evaluate, dark sync serve binds all interfaces unauthenticated. Accepted for a local-first v1 (a
  peer is your own instance or a trusted collaborator on a private tailnet), marked CLEANUP(sync-security).
  Already hardened in this PR: the arbitrary-path sqlite* builtins are capability-gated (Needs.fileReadWrite
  — the interpreter no longer skips their check), and syncBlobInsert re-hashes peer bytes so a poisoned/
  mismatched blob is rejected. Remaining: verify op ids against content, reject future-dated origin_ts, gate
  httpGetUnsafeBytes away from synced-value evaluation (TODO(sync-ssrf)), bind localhost + expose via dark devices serve, authenticated serving.
• Structural convergence — residual. Concurrent rebases of one branch now converge (synced origin_ts,
  LWW-gated via base_ts/base_op; merge/archive are monotonic set-once). Residual: a same-millisecond rebase
  tie-breaks by op-id but no SyncConflict is recorded for structural divergence yet (name conflicts are).
• Sync bandwidth — wholesale. Sync now transfers every branch's ops + content. Fine for a personal
  tailnet; a future per-branch scoped sync (server pre-filtering by what a puller wants) would cut the transfer
  when it returns.
• Manage the fold from Dark eventually. The tables are already Dark-readable at native speed via
  Stdlib.Sqlite (spike: ~10.8k ops in ~2.26s). The fold stays native F# only because per-row Dark access is
  still ~10ms/Dict.get (folding the seed in-interpreter is ~9 min). When that interpreter cost drops the fold
  moves to Dark over the same tables — gated on interpreter perf, not architecture.
• Smaller. Give receiveOps/eventsSince distinct commit/event record types (two identical tuples today);
  consolidate the Sqlite.fs stream/record boilerplate + nits (shared conflict-status constants, short() dupes
  hashToShort, a stale Sqlite.fs header, double-deserialize in receiveOps); finish chosen→choice on the
  Dark/wire mirrors; align the daemon config namespace (sync.daemon.* vs apps.sync.*) and the table suffixes.

---

This is aimed squarely at a local-first v1 — a peer is your own instance, or a trusted collaborator on a
private tailnet. The trust model (first TODO) is the one thing I'd want hardened before this faces untrusted
peers. Everything above is working and tested; I've flagged the residual edges honestly rather than hide them,
and the biggest ones (structural-conflict recording, proper per-branch scoped sync) are noted as follow-ups
that don't block the local-first case.