sapientinc/HRM-Text-1B · Hugging Face

Source: https://huggingface.co/sapientinc/HRM-Text-1B

A 1 B-parameter language model checkpoint built on the**Hierarchical Reasoning Model (HRM)**architecture, trained by Sapient Intelligence from scratch on structured public datasets.

HRM is a dual-timescale recurrent architecture: two Transformer modules (H = high-level / slow, L = low-level / fast) iterate over the same input embeddings forH\_cycles × \(L\_cycles \+ 1\)steps, with additive state injection (z\_L \+ z\_H). This gives effectively unbounded compute depth at bounded parameter count.

https://huggingface.co/sapientinc/HRM-Text-1B#disclaimerDisclaimer

This is apre-alignmentmodel checkpoint, not a chat or instruction-following assistant. It is pre-trained on a PrefixLM objective with condition prefix tokens and hasnotbeen multi-turn dialogue tuned, long-context adapted, instruction-tuned, RLHF-trained, or otherwise aligned for assistant-style use. If you want to use HRM-Text like a chat model, you would need to perform further alignment, such as SFT and/or RL, on task-specific data. This checkpoint is meant to serve as a starting point, not a finished assistant.

Practical guidance for prompting the raw checkpoint:

• NLP tasks (classification, extraction, structured output, short-form QA): use thedirectcondition with 2–8 few-shot in-context examples.direct+ few-shot is the strongest zero-extra-training setup we have measured; pure zero-shot is noticeably weaker.
• Reasoning / math / open-ended generation: use thecomposite conditionsynth,cot. This isonecomposite prefix, not two alternatives — at tokenization time the comma-separated tags are mapped to their prefix tokens and concatenated, in order, into a single prefix block. Sosynth,cotproduces the two-token prefix<\|quad\_end\|\><\|object\_ref\_end\|\>(synth first, then cot), wrapped in the usual<\|im\_start\|\>…<\|im\_end\|\>envelope. Under this composite the model exhibits some chain-of-thought / instruct-like behavior — enough to answer many zero-shot math and reasoning prompts in a step-by-step style — but quality is uneven and below an instruction-tuned model of comparable size. Treat this “instruct” ability as a side effect of the pre-training mix, not a guaranteed capability.

The four single condition tags and their assigned tokenizer special tokens (token names are legacy implementation details; you can compose any subset, comma-separated, in the order you want them emitted):

• direct→<\|object\_ref\_start\|\>— direct answer, no CoT
• cot→<\|object\_ref\_end\|\>— chain-of-thought
• noisy→<\|quad\_start\|\>— noisy / web-crawl style
• synth→<\|quad\_end\|\>— synthetic / curated style

https://huggingface.co/sapientinc/HRM-Text-1B#requirementsRequirements

Use a Transformers build that includes thehrm\_textmodel class. If your installed release does not include it yet, install Transformers directly from the upstreammainbranch:

pip install --upgrade "git+https://github.com/huggingface/transformers.git@main"

https://huggingface.co/sapientinc/HRM-Text-1B#model-detailsModel details

FieldValueParameters~1 BHidden size1536Layers (per H / L stack)16Attention heads12 (MHA, head_dim 128)H_cycles × L_cycles2 × 3Max sequence length4096Vocabulary65,536EmbeddingScaled (lecun_normal)Position encodingRoPE (theta 10000)ActivationSwiGLUNormalizationParameterless Pre-RMSNormAttentionGated (sigmoid output gate)Training unique tokens40 BOptimizerAdamATan2 (beta 0.9 / 0.95, wd 0.1, EMA 0.9999)LR2.2e-4 (warmup 2000 steps)Global batch196,608 tokensdtypebfloat16

https://huggingface.co/sapientinc/HRM-Text-1B#usageUsage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "sapientinc/HRM-Text-1B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    trust_remote_code=True,
).cuda().eval()

# synth,cot composite — reasoning / CoT style (see Disclaimer for other modes)
condition = "<|quad_end|><|object_ref_end|>"
prompt = f"<|im_start|>{condition}Explain why the sky is blue.<|im_end|>"

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
# Mark the prompt as a single bidirectional prefix block — see "PrefixLM mask" below.
inputs["token_type_ids"] = torch.ones_like(inputs["input_ids"])

with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(out[0], skip_special_tokens=False))

https://huggingface.co/sapientinc/HRM-Text-1B#prefixlm-mask–pass-token_type_idsPrefixLM mask — passtoken\_type\_ids

HRM-Text was pre-trained with a PrefixLM mask: prompt tokens attend bidirectionally to each other, response tokens attend causally. To match the training-time forward at inference you must tell the model which positions are prefix.

In the current Transformers port the mask is controlled bytoken\_type\_ids:

• token\_type\_ids\[i\] == 1→ positioniis part of the prefix block (bidirectional within the block).
• otherwise → causal.

If you omittoken\_type\_ids, attention falls back topure causal, which doesnotmatch the pre-training distribution and will give noticeably worse logits. The simplest correct call passestoken\_type\_ids = torch\.ones\_like\(input\_ids\), marking the entire input prompt as one bidirectional prefix block — exactly how training-time prefill ran.

https://huggingface.co/sapientinc/HRM-Text-1B#architectureArchitecture

The recurrent core (per forward pass, in inference mode):

z_H = embed(input_ids) * embedding_scale
z_L = z_L_init.expand_as(z_H)

for _ in range(H_cycles):
    for _ in range(L_cycles):
        z_L = L_module(z_L + z_H)
    z_H = H_module(z_H + z_L)
return z_H

Both stacks share the same Transformer block design (gated attention, RoPE, SwiGLU, pre-RMSNorm); see Model details above for shapes.

https://huggingface.co/sapientinc/HRM-Text-1B#training-dataTraining data

Pre-trained on a sampled mixture of publicly available text corpora. The full dataset composition, sampling weights, and preprocessing pipeline are open-sourced:

https://huggingface.co/sapientinc/HRM-Text-1B#limitationsLimitations

• English only (training corpus is predominantly English).
• HRM-Text-1B was not trained on code datasets, therefore its rather weak performance on coding tasks was expected. Early third-party code SFT experiments on roughly 1B tokens of code data improved coding benchmark scores from low single digits to around 40–50, suggesting promising adaptation potential, but those results are not part of this checkpoint.
• Outputs may vary under different environments, and may contain inaccuracies, biases, or unsafe contents.

https://huggingface.co/sapientinc/HRM-Text-1B#licenseLicense

Apache License 2.0.

https://huggingface.co/sapientinc/HRM-Text-1B#citationCitation

Citation information will be added with the accompanying paper.