brooktrails/gllm
Watch
2
0
Fork
You've already forked gllm
 0
Code Issues 3 Pull requests 2 Projects Releases Packages Wiki Activity Actions

feat(config): v4 rope_scaling, per-layer-type rope, and layer_types #115

Merged
rcsheets merged 2 commits from feat/config-rope-scaling-layer-types into main 2026-10-04 06:15:28 +00:00
Conversation 1 Commits 2 Files changed 4 +269 -3
rcsheets commented 2026-10-04 05:25:01 +00:00
Owner
Copy link

config.Load had two gaps in how it reads rope settings, both silent:

  • rope_scaling was ignored. transformers v4 checkpoints carry their rope scaling there, with theta at the top level. A YaRN checkpoint written that way got plain rope and no error. allenai/Olmo-3-7B-Think is one.
  • Rope that differs by layer could not be represented. OLMo 3 gives its sliding-window layers plain rope and its full-attention layers YaRN. transformers 5 writes that as rope_parameters keyed by layer type ({"sliding_attention": {...}, "full_attention": {...}}). The flat decode reads that shape as all zeros, and a zero theta produces NaN logits.

What changes

  • rope_scaling is folded into the v5 fields by foldRopeScaling, following transformers' per-model conversion:

    • by default it becomes RopeParameters, with the top-level theta;
    • for OLMo 3 it becomes the full_attention entry of RopeByLayerType, with plain rope for sliding_attention (as in Olmo3Config.convert_rope_params_to_dict).

    The old type key is accepted for rope_type.

  • Nested rope_parameters (keyed by layer type) are recognized from the raw JSON into RopeByLayerType, at the top level and under text_config.

  • layer_types is read, and Load rejects a list whose length differs from num_hidden_layers.

  • Accessors for models:

    • RopeFreqsForLayer(l, headDim) returns layer l's rope, falling back to RopeFreqs when layers share one.
    • SlidingWindowFor(l) windows only sliding_attention layers when layer_types is present. Without it, sliding_window applies to every layer, as Mistral 7B v0.1 used it. 0 means unlimited.

    layer_types' vocabulary differs by family (GLM uses it for its own attention kinds), so only these accessors interpret it.

  • AGENTS.md describes the three spellings and the per-model fold.

No model reads the new accessors yet; mistral.go still rejects sliding_window. They are for the OLMo 3 architecture, which follows. No checkpoint gllm serves today uses rope_scaling or sliding layer types, so their rope is unchanged.

Verification

  • go build ./..., go vet ./..., go test ./... pass; gofmt is clean.
  • TestOlmo3RopeByLayerType loads OLMo 3's config in both spellings, v4 rope_scaling as published and v5 keyed rope_parameters. It checks inv_freq at six pairs and the attention scaling for sliding and full layers against transformers 5.14.1's Olmo3RotaryEmbedding on the real checkpoint: plain rope with scaling 1, and YaRN with 1.2079441541679836.
  • TestRopeScalingLegacy: a rope_scaling with the old type key on a single-rope model yields YaRN RopeParameters with the top-level theta.
  • TestSlidingWindowFor: per-layer windows from layer_types, a window on every layer without it, and none without sliding_window.
  • TestLayerTypesLength: a mismatched layer_types length is rejected.
  • Every checkpoint on the development box (Granite 4.2, Ministral 3, Mistral Small 4 / Medium 3.5, Nemotron 3, GLM 5.3, both OLMo 3 checkpoints) loads. The two OLMo 3 checkpoints report plain rope and a 4096 window on layer 0, and YaRN with scaling 1.2079442 on layer 31.

🤖 Generated with Claude Code

`config.Load` had two gaps in how it reads rope settings, both silent: - **`rope_scaling` was ignored.** transformers v4 checkpoints carry their rope scaling there, with theta at the top level. A YaRN checkpoint written that way got plain rope and no error. `allenai/Olmo-3-7B-Think` is one. - **Rope that differs by layer could not be represented.** OLMo 3 gives its sliding-window layers plain rope and its full-attention layers YaRN. transformers 5 writes that as `rope_parameters` keyed by layer type (`{"sliding_attention": {...}, "full_attention": {...}}`). The flat decode reads that shape as all zeros, and a zero theta produces NaN logits. ## What changes - **`rope_scaling`** is folded into the v5 fields by `foldRopeScaling`, following transformers' per-model conversion: - by default it becomes `RopeParameters`, with the top-level theta; - for OLMo 3 it becomes the `full_attention` entry of `RopeByLayerType`, with plain rope for `sliding_attention` (as in `Olmo3Config.convert_rope_params_to_dict`). The old `type` key is accepted for `rope_type`. - **Nested `rope_parameters`** (keyed by layer type) are recognized from the raw JSON into `RopeByLayerType`, at the top level and under `text_config`. - **`layer_types`** is read, and Load rejects a list whose length differs from `num_hidden_layers`. - **Accessors for models:** - `RopeFreqsForLayer(l, headDim)` returns layer `l`'s rope, falling back to `RopeFreqs` when layers share one. - `SlidingWindowFor(l)` windows only `sliding_attention` layers when `layer_types` is present. Without it, `sliding_window` applies to every layer, as Mistral 7B v0.1 used it. 0 means unlimited. `layer_types`' vocabulary differs by family (GLM uses it for its own attention kinds), so only these accessors interpret it. - AGENTS.md describes the three spellings and the per-model fold. No model reads the new accessors yet; `mistral.go` still rejects `sliding_window`. They are for the OLMo 3 architecture, which follows. No checkpoint gllm serves today uses `rope_scaling` or sliding layer types, so their rope is unchanged. ## Verification - `go build ./...`, `go vet ./...`, `go test ./...` pass; gofmt is clean. - `TestOlmo3RopeByLayerType` loads OLMo 3's config in both spellings, v4 `rope_scaling` as published and v5 keyed `rope_parameters`. It checks inv_freq at six pairs and the attention scaling for sliding and full layers against transformers 5.14.1's `Olmo3RotaryEmbedding` on the real checkpoint: plain rope with scaling 1, and YaRN with 1.2079441541679836. - `TestRopeScalingLegacy`: a `rope_scaling` with the old `type` key on a single-rope model yields YaRN `RopeParameters` with the top-level theta. - `TestSlidingWindowFor`: per-layer windows from `layer_types`, a window on every layer without it, and none without `sliding_window`. - `TestLayerTypesLength`: a mismatched `layer_types` length is rejected. - Every checkpoint on the development box (Granite 4.2, Ministral 3, Mistral Small 4 / Medium 3.5, Nemotron 3, GLM 5.3, both OLMo 3 checkpoints) loads. The two OLMo 3 checkpoints report plain rope and a 4096 window on layer 0, and YaRN with scaling 1.2079442 on layer 31. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
feat(config): v4 rope_scaling, per-layer-type rope, and layer_types
All checks were successful
ci / test_and_build (pull_request) Successful in 49s
Details
3d2b2fe6a7 
config.Load ignored rope_scaling, the transformers v4 key for rope
scaling, so a YaRN checkpoint written that way got plain rope with no
error. It also could not represent models whose layers differ in rope:
OLMo 3 gives its sliding-window layers plain rope and its full-attention
layers YaRN, which transformers v5 writes as rope_parameters keyed by
layer type -- a shape the flat decode read as all zeros (theta 0).

Load now folds rope_scaling into RopeParameters (or, for OLMo 3, into
the full-attention entry of RopeByLayerType, as Olmo3Config does),
recognizes the nested rope_parameters form, accepts the old "type" key,
and reads layer_types. Models get per-layer settings through
RopeFreqsForLayer and SlidingWindowFor. No checkpoint gllm serves today
uses rope_scaling or sliding layer types, so their rope is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pr-reviewer-bot commented 2026-10-04 05:25:03 +00:00
Collaborator
Copy link

Automated review by pr-reviewer v0.54.0 | Safety Check | Nemotron 3 Nano | tracking id r-c1ee4f-77606c
This is an AI-generated review and may contain mistakes.

Status: ❌ Failed

Review failed. Tracking id r-c1ee4f-77606c — see logs for details.

Comment @pr-reviewer-bot retry to try again.

<!-- pr-reviewer:review --> *Automated review by [pr-reviewer](https://git.brooktrails.org/brooktrails/pr-reviewer) v0.54.0 | Safety Check | Nemotron 3 Nano | tracking id `r-c1ee4f-77606c`* *This is an AI-generated review and may contain mistakes.* **Status:** ❌ Failed --- Review failed. Tracking id `r-c1ee4f-77606c` — see logs for details. Comment `@pr-reviewer-bot retry` to try again.
Merge remote-tracking branch 'origin/main' into feat/config-rope-scaling-layer-types
All checks were successful
ci / test_and_build (pull_request) Successful in 46s
Details
c26d9ad1a3 
# Conflicts:
#	internal/config/config.go
rcsheets deleted branch feat/config-rope-scaling-layer-types 2026-10-04 06:15:33 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
brooktrails/gllm!115
No description provided.