Local Weights · curator journey · concept
A curator selects models and runtime settings for a team. This document explores tools for comparing candidates, recording benchmark evidence, editing tiers, and publishing signed releases. The benchmark and Radicle COB workflows are proposals; the maturity tags distinguish current code from possible future work.
A profile tier identifies the model, runtime artifact, arguments, environment, and context size. Benchmark results are useful only when they refer to that exact configuration. If the pinned runtime artifact also contains llama-bench, the manager could run the matching benchmark tool and record the result.
tested unit = (model sha256 · artifact build · argv · env · context) → signed benchmark result → evidence considered by the curator → signed profile release
Step 1 · compare candidates
The curator needs a model that fits several hardware classes. A comparison view can show throughput, peak memory, configuration, and the number of submitted runs. Results with limited evidence should be clearly marked.
implementation — proposed benchmark COBs speculative would live in the registry's Radicle repository and be signed by each submitter's DID. The dashboard could aggregate local copies of those records and show medians, sample counts, and raw submissions.
| Tested unit | tok/s | peak mem | evidence | |
|---|---|---|---|---|
| Qwen3.7-32B · IQ4_XSb10112 · metal · fa=on · kv q8_0 · 128k | 44.2 | 24.9 GB | 14 nodes · 31 runs | fits |
| Qwen3.6-35B · IQ4_XSb10034 · metal · defaults · 128k | 38.7 | 25.4 GB | 9 nodes · 22 runs | fits · current |
| Devstral-2-30B · Q4_K_Mb10112 · metal · fa=on · 96k | 41.0 | 27.8 GB | 2 nodes · 3 runs | thin evidence |
| GLM-5-Air · MXFP4b10112 · metal · fa=on · 128k | 47.5 | 31.2 GB | 6 nodes · 11 runs | no headroom @32 GB |
Step 2 · run a benchmark
The curator acquires the candidate through the normal verified path, runs llama-bench from the pinned runtime artifact, reviews the result, and can choose to publish it. A separate request workflow could ask other machines to run the same test.
implementation — acquisition and verification today; running llama-bench from a pinned artifact near; measurement requests and unattended fleet scheduling speculative. A result would include the tested configuration and the hardware facts already collected by the tier matcher.
Pinning the configuration makes the run repeatable. Results can still vary with hardware state, software outside the bundle, and benchmark conditions, so raw runs remain available for review.
Step 3 · authoring
The editor shows the selected model, runtime, context, arguments, and environment next to available fit and performance results. It uses the same schema validation as the signed release tooling.
implementation — schema validation, reserved-flag checks, and memory-range checks today; estimated memory fit calibrated against measured peak RSS near; evidence sidebar speculative.
Step 4 · release
The release view shows the changes by hardware tier, the evidence used, and the note developers will see. The curator reviews the diff before signing a new rollback-protected release version.
implementation — signing, versioning, and rollback protection today; a release-note field near; publishing through Radicle and linking evidence COBs speculative.
| Tier | Change | Evidence |
|---|---|---|
| apple-silicon-64gb-plus | Qwen3.6 Q4_K_M → Qwen3.7 Q4_K_M · fa=on, kv q8_0 | 11 nodes |
| apple-silicon-32-to-63gb | Qwen3.6 IQ4_XS → Qwen3.7 IQ4_XS · +14% tok/s | 14 nodes |
| linux-strix-halo-64gb | unchanged | 5 nodes |
Step 5 · review the release
After release, the curator can review update status, diagnostics shared by developers, and benchmark results for the released configuration. Differences by hardware class can become inputs to the next revision.
implementation — copyable diagnostics near; report COBs, adoption reporting, and a regression view speculative. Reports should omit prompts, source code, local paths, and hardware serial numbers.
| Signal | Detail | |
|---|---|---|
| adoption | 9 of 11 fleet machines updated · 7 via LAN, 2 via HF | |
| regression | m1-max · 64 GB: tg128 down 22% vs release evidence (2 nodes agree) | view COBs |
| report | 1 × port conflict (foreign process) · repair succeeded after reboot | view |
The M1 Max result can be investigated and, if confirmed, added as a test case for the next release.
Publishes a profile for hardware or workflows they know well.
same schema · same signing · same verification · same app
Maintains approved profiles for managed machines.
same schema · same signing · same verification · same app
| Data | Use | Limits and checks |
|---|---|---|
| Benchmark results (tok/s, peak memory, TTFB) | useful | Pin the tested configuration and tool, retain raw runs, show sample counts, and separate submitters. A signature identifies the submitter; it does not prove the result is accurate. |
| Hardware compatibility (for example, OOM on 32 GB) | useful | Record the exact configuration and hardware class. Confirm important results with more than one machine where possible. |
| Evaluation scores | limited | Comparable only when the harness, dataset, prompts, and scoring version are pinned. Results may be expensive to reproduce and easier to optimize selectively. |
| Argument and tuning proposals | patches | Submit changes as repository patches with benchmark or compatibility results. The curator reviews and decides whether to include them. |
| Selection, signing, and release notes | curator | Community results are inputs. The curator remains responsible for the selected profile and signs the release. |