Add SmallReason-ColBERT-32M: base + 129-param importance head

1_Dense/config.json

@@ -0,0 +1 @@
+{"in_features": 384, "out_features": 768, "bias": false, "activation_function": "torch.nn.modules.linear.Identity", "use_residual": false}

1_Dense/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fd2aa41a2974566861ff9690be4f4c2b9a917ee0f97e338911bd66fa06dafddd
+size 589912

2_Dense/config.json

@@ -0,0 +1 @@
+{"in_features": 768, "out_features": 768, "bias": false, "activation_function": "torch.nn.modules.linear.Identity", "use_residual": true}

2_Dense/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ec4fecf007916948b029915b76dbf37efd5241a21a497dfb4f2abe622410b55a
+size 1179736

3_Dense/config.json

@@ -0,0 +1 @@
+{"in_features": 768, "out_features": 128, "bias": false, "activation_function": "torch.nn.modules.linear.Identity", "use_residual": false}

3_Dense/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c99e270c45b0db9b5ffbf5e0de10ba729680aa778807b2329189b682d686203d
+size 196696

README.md

@@ -0,0 +1,240 @@
+---
+language:
+- en
+tags:
+- ColBERT
+- PyLate
+- sentence-transformers
+- sentence-similarity
+- feature-extraction
+- late-interaction
+- reasoning-retrieval
+- edge
+- loss:CachedContrastive
+base_model: mixedbread-ai/mxbai-edge-colbert-v0-32m
+datasets:
+- reasonir/reasonir-data
+- hanhainebula/bge-reasoner-data
+pipeline_tag: sentence-similarity
+library_name: PyLate
+license: cc-by-nc-4.0
+---
+
+# SmallReason-ColBERT (32M)
+
+An ultra-small late-interaction retriever for **reasoning-intensive** retrieval.
+32M parameters, plus a **129-parameter query-side importance head**.
+
+**21.41 mean nDCG@10 on BRIGHT** — above every ≤33M ColBERT we evaluated, and within
+1.21 of the 4.7× larger 150M Reason-ModernColBERT.
+
+---
+
+## ⚠ Read this before loading
+
+This model is a ColBERT base **plus a small importance head** stored in
+`importance_head/`. The head is *not* part of `modules.json`, so a standard PyLate /
+sentence-transformers load **silently ignores it** and gives you the un-headed base:
+
+| How you load it | What you get | BRIGHT mean |
+|---|---|---:|
+| `pylate.models.ColBERT(...)` — plain load | base only, head ignored, **no error** | 19.61 |
+| `WeightedColBERT.from_base(...)` — see below | full model | **21.41** |
+
+There is no warning when the head is skipped. If you are reproducing the paper number,
+use the second path.
+
+---
+
+## Usage
+
+The loader is a single file, [`weighted_colbert.py`](https://github.com/DataScience-UIBK/SmallReason-ColBERT/blob/main/src/weighted_colbert.py),
+from the companion repository.
+
+```python
+from weighted_colbert import WeightedColBERT
+
+model = WeightedColBERT.from_base(
+    "DataScience-UIBK/SmallReason-ColBERT-32M",   # auto-detects importance_head/
+    query_length=256,
+    document_length=2048,
+    device="cuda:0",
+)
+
+queries = ["What factors affect the number of Hadley cells a planet has, and how?"]
+docs = [
+    "Hadley cells are driven by differential solar heating; their number scales with "
+    "planetary rotation rate and atmospheric depth.",
+    "The best pasta recipe uses semolina flour and plenty of salted boiling water.",
+]
+
+q_embs, q_weights = model.encode(queries, is_query=True, return_weights=True)
+d_embs = model.encode(docs, is_query=False)
+
+for i, d in enumerate(d_embs):
+    score = WeightedColBERT.weighted_maxsim(q_embs[0], q_weights[0], d)
+    print(i, float(score))
+```
+
+`weighted_maxsim` implements the evaluation-time score
+
+$$s(q,d) = \frac{\sum_t w_t \cdot \max_j \mathbf{Q}_t \cdot \mathbf{D}_j}{\sum_t w_t}$$
+
+where $w_t = \sigma(\mathbf{W}\mathbf{Q}_t + b)$ is the learned per-query-token gate.
+The `1/\sum_t w_t` factor is constant across documents for a fixed query, so it does not
+change ranking — it only keeps scores comparable across queries of different length.
+
+### Base only (no head)
+
+If you want the reasoning-tuned base without the gate (19.61 on BRIGHT), load it as an
+ordinary PyLate ColBERT — the head files are simply unused:
+
+```python
+from pylate import models
+base = models.ColBERT("DataScience-UIBK/SmallReason-ColBERT-32M",
+                      query_length=256, document_length=2048)
+```
+
+---
+
+## Results
+
+### BRIGHT (nDCG@10 ×100)
+
+Evaluated with brute-force MaxSim, `query_length=256` (Pony: 32), `document_length=2048`.
+
+| Split | upstream 32M | base (no head) | **SmallReason-ColBERT** |
+|---|---:|---:|---:|
+| biology | 28.70 | 33.16 | **34.17** |
+| earth_science | 42.29 | 44.28 | **45.03** |
+| economics | 17.65 | **20.25** | 19.99 |
+| psychology | 21.93 | 24.91 | **24.94** |
+| robotics | 18.09 | **18.65** | 18.14 |
+| stackoverflow | 16.49 | 16.66 | **17.21** |
+| sustainable_living | 18.64 | 20.11 | **21.07** |
+| pony | 12.90 | **22.77** | 19.33 |
+| leetcode | 16.15 | 17.40 | **29.98** |
+| aops | 9.80 | 4.89 | **10.29** |
+| theoremqa_questions | 12.51 | 9.04 | **13.00** |
+| theoremqa_theorems | 2.76 | 3.19 | **3.74** |
+| **Mean** | 18.16 | 19.61 | **21.41** |
+
+The head is worth **+1.80** mean nDCG@10 over the same base, concentrated in the long,
+symbol-dense splits: LeetCode +12.58, AoPS +5.40, TheoremQA-questions +3.96.
+
+### Reference points
+
+| Model | Params | BRIGHT mean |
+|---|---:|---:|
+| **SmallReason-ColBERT** | **32M** | **21.41** |
+| answerai-colbert-small-v1 | 33M | 18.49 |
+| mxbai-edge-colbert-v0-17m | 17M | 18.60 |
+| GTE-ModernColBERT-v1 | 150M | 21.72 |
+| Reason-ModernColBERT | 150M | 21.97 (our protocol) / 22.62 (published) |
+
+### NanoBEIR sanity (classical IR)
+
+The gate is trained on long reasoning queries, so it is expected to give a little back
+on short keyword queries. It does, but not much:
+
+| Model | All 13 | Excl. Touche-2020 |
+|---|---:|---:|
+| upstream 32M | 60.47 | 65.51 |
+| base (no head) | 60.93 | 65.35 |
+| **SmallReason-ColBERT** | 60.00 | 65.00 |
+
+---
+
+## How it works
+
+Three stages, on top of `mixedbread-ai/mxbai-edge-colbert-v0-32m`:
+
+1. **Widen the projection** 64 → 128 dims. The first 64 rows are inherited; the new 64
+   are initialised from `N(0, σ²)` with `σ` at 10% of the original weight-matrix std —
+   small enough to leave MaxSim ≈ unchanged at step 0, non-zero so the new channels
+   actually receive gradient.
+2. **Two-stage base training** — a varied-length warmup on ReasonIR-VL, then a
+   hard-negative polish on merged ReasonIR-HQ + BGE-Reasoner. Both stages use PyLate's
+   `CachedContrastive` loss over in-batch negatives.
+3. **Importance head** — freeze the base, train a single `Linear(128, 1)` + sigmoid
+   (129 parameters) to weight each query token.
+
+### The one non-obvious trick
+
+The head is **trained against the un-normalised** weighted score
+`Σ w_t · max_j(Q_t · D_j)` but **evaluated against the length-normalised** one.
+
+This asymmetry is the single most consequential choice in the recipe. Train against the
+normalised score instead and the per-pair score difference is bounded by one token's
+cosine range, the cross-entropy gradient collapses, the loss stalls near `ln 2`, the
+gates never leave their initialisation — and BRIGHT drops by **3.59** nDCG@10.
+
+The head is initialised `W = 0`, `b = 5`, so every gate starts at `σ(5) ≈ 0.993` and the
+head is a no-op against the frozen base at step zero.
+
+### What the head actually learns
+
+Not soft-IDF. Across ~199K BRIGHT query tokens the gate–IDF Spearman correlation is
+**ρ = −0.02** — statistically detectable, practically zero. Per-split mean gate sits in
+0.43–0.47 with std ≈ 0.10: the head is a soft re-weighting, not a selector. A fixed IDF
+gate on the same base reaches only 20.06, against 21.41 for the learned head.
+
+---
+
+## Training
+
+| | Warmup | Polish | Head |
+|---|---|---|---|
+| Data | ReasonIR-VL (~245K) | merged ReasonIR-HQ + BGE-Reasoner (~2.7M) | same merged set |
+| Loss | CachedContrastive | CachedContrastive | CE over `[s_pos, s_neg]` |
+| LR | 1e-5 | 5e-6 | 5e-4 (AdamW, wd=0) |
+| Batch | 32/GPU × accum 4 × 8 GPU | 32/GPU × accum 2 × 8 GPU | 16 triples/step, 1 GPU |
+| Steps | 1 epoch (~8 h) | 1 epoch (~16 h) | 3,000 steps (~12 min) |
+| Lengths | q 256 / doc 2048 | q 256 / doc 2048 | q 256 / doc 2048 |
+| Precision | bf16 + FA2 | bf16 + FA2 | fp32 head, frozen bf16 base |
+
+Base training: 8× H100 across two nodes, ~24 h total. Head training: one H100, ~12 min.
+
+---
+
+## Limitations
+
+- **Scale.** The recipe was developed and validated at 32M. It does not transfer for
+  free — the same head at 17M gives **no** gain.
+- **Frozen base.** The head is trained on a frozen base; joint fine-tuning is unexplored.
+- **Late-interaction cost.** The head is nearly free, but the model still carries
+  multi-vector storage and scoring costs. The efficiency claim is about parameter count,
+  not about matching single-vector retrieval.
+- **Short queries.** Pony (32-token queries) regresses relative to the un-headed base —
+  a per-token gate needs tokens to discriminate between.
+- **Oblique queries.** On OBLIQ-Bench (stance / intent / tip-of-the-tongue) the model is
+  near zero (mean 3.66) and is beaten by every baseline there. Reported as a deliberate
+  negative result; embedding similarity is the wrong tool for that class of query.
+- **Synthetic teacher data.** Training data is synthetic with cross-encoder-mined hard
+  negatives; biases in that mining can propagate.
+
+## License
+
+**CC-BY-NC-4.0**, inherited from the ReasonIR and BGE-Reasoner training data.
+The upstream base model (`mixedbread-ai/mxbai-edge-colbert-v0-32m`) is Apache-2.0, and
+the companion training/inference **code** is released under Apache-2.0 — but these
+**weights** are non-commercial.
+
+## Citation
+
+```bibtex
+@inproceedings{smallreason-colbert,
+  title     = {SmallReason-ColBERT: An Ultra-Small Late-Interaction Retriever
+               for Reasoning Intensive Retrieval},
+  author    = {TBD},
+  booktitle = {Proceedings of EMNLP},
+  year      = {2026}
+}
+```
+
+## Acknowledgements
+
+Thanks to Antoine Chaffin (LightOn, Reason-ModernColBERT) for flagging the upstream
+`2_Dense/use_residual` config bug in `mxbai-edge-colbert-v0-32m` — the base weights were
+trained with a residual on that layer while the shipped config said otherwise. This
+model uses the patched config (`use_residual: true`).

config.json

@@ -0,0 +1,50 @@
+{
+  "_name_or_path": "/data/fs201059/aa17626/modernTransformer/repro_reason_moderncolbert/output_edge32m_d128_fixed/stage-vl/final",
+  "architectures": [
+    "ModernBertModel"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 50281,
+  "causal_mask": false,
+  "classifier_activation": "gelu",
+  "classifier_bias": false,
+  "classifier_dropout": 0.0,
+  "classifier_pooling": "mean",
+  "cls_token_id": 50281,
+  "decoder_bias": true,
+  "deterministic_flash_attn": false,
+  "dtype": "float32",
+  "embedding_dropout": 0.0,
+  "eos_token_id": 50282,
+  "global_attn_every_n_layers": 3,
+  "global_rope_theta": 160000.0,
+  "gradient_checkpointing": false,
+  "hidden_activation": "gelu",
+  "hidden_size": 384,
+  "initializer_cutoff_factor": 2.0,
+  "initializer_range": 0.02,
+  "intermediate_size": 576,
+  "is_causal": false,
+  "layer_norm_eps": 1e-05,
+  "local_attention": 128,
+  "local_rope_theta": 160000.0,
+  "max_position_embeddings": 7999,
+  "mlp_bias": false,
+  "mlp_dropout": 0.0,
+  "model_type": "modernbert",
+  "norm_bias": false,
+  "norm_eps": 1e-05,
+  "num_attention_heads": 6,
+  "num_hidden_layers": 10,
+  "pad_token_id": 50283,
+  "position_embedding_type": "sans_pos",
+  "reference_compile": false,
+  "repad_logits_with_grad": false,
+  "sep_token_id": 50282,
+  "sparse_pred_ignore_index": -100,
+  "sparse_prediction": false,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.48.2",
+  "vocab_size": 50370
+}

config_sentence_transformers.json

@@ -0,0 +1,21 @@
+{
+  "model_type": "ColBERT",
+  "__version__": {
+    "sentence_transformers": "4.0.2",
+    "transformers": "4.48.2",
+    "pytorch": "2.5.1"
+  },
+  "prompts": {
+    "query": "",
+    "document": ""
+  },
+  "default_prompt_name": null,
+  "similarity_fn_name": "MaxSim",
+  "query_prefix": "[Q] ",
+  "document_prefix": "[D] ",
+  "query_length": 256,
+  "document_length": 2048,
+  "attend_to_expansion_tokens": false,
+  "skiplist_words": [],
+  "do_query_expansion": false
+}

importance_head/config.json

@@ -0,0 +1,5 @@
+{
+  "in_features": 128,
+  "head_hidden": 0,
+  "init_kind": "sigmoid_bias5"
+}

importance_head/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:95f2301992822060bf0c6ad18ceb993174d752266fe51fbba799f4e0e534e09f
+size 660

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7d86cdb2c133be712ea760ea2c053d0eed2dc712d99981ecef4d83eb1827d95b
+size 63773752

modules.json

@@ -0,0 +1,26 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.models.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Dense",
+    "type": "pylate.models.Dense.Dense"
+  },
+  {
+    "idx": 2,
+    "name": "2",
+    "path": "2_Dense",
+    "type": "pylate.models.Dense.Dense"
+  },
+  {
+    "idx": 3,
+    "name": "3",
+    "path": "3_Dense",
+    "type": "pylate.models.Dense.Dense"
+  }
+]

sentence_bert_config.json

@@ -0,0 +1,4 @@
+{
+  "max_seq_length": 255,
+  "do_lower_case": true
+}

special_tokens_map.json

@@ -0,0 +1,31 @@
+{
+  "cls_token": {
+    "content": "[CLS]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "[MASK]",
+    "lstrip": true,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "[MASK]",
+  "sep_token": {
+    "content": "[SEP]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "[UNK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,968 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "|||IP_ADDRESS|||",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "1": {
+      "content": "<|padding|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50254": {
+      "content": "                        ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50255": {
+      "content": "                       ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50256": {
+      "content": "                      ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50257": {
+      "content": "                     ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50258": {
+      "content": "                    ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50259": {
+      "content": "                   ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50260": {
+      "content": "                  ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50261": {
+      "content": "                 ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50262": {
+      "content": "                ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50263": {
+      "content": "               ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50264": {
+      "content": "              ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50265": {
+      "content": "             ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50266": {
+      "content": "            ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50267": {
+      "content": "           ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50268": {
+      "content": "          ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50269": {
+      "content": "         ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50270": {
+      "content": "        ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50271": {
+      "content": "       ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50272": {
+      "content": "      ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50273": {
+      "content": "     ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50274": {
+      "content": "    ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50275": {
+      "content": "   ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50276": {
+      "content": "  ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50277": {
+      "content": "|||EMAIL_ADDRESS|||",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50278": {
+      "content": "|||PHONE_NUMBER|||",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50279": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50280": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50281": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50282": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50283": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50284": {
+      "content": "[MASK]",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50285": {
+      "content": "[unused0]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50286": {
+      "content": "[unused1]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50287": {
+      "content": "[unused2]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50288": {
+      "content": "[unused3]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50289": {
+      "content": "[unused4]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50290": {
+      "content": "[unused5]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50291": {
+      "content": "[unused6]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50292": {
+      "content": "[unused7]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50293": {
+      "content": "[unused8]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50294": {
+      "content": "[unused9]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50295": {
+      "content": "[unused10]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50296": {
+      "content": "[unused11]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50297": {
+      "content": "[unused12]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50298": {
+      "content": "[unused13]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50299": {
+      "content": "[unused14]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50300": {
+      "content": "[unused15]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50301": {
+      "content": "[unused16]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50302": {
+      "content": "[unused17]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50303": {
+      "content": "[unused18]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50304": {
+      "content": "[unused19]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50305": {
+      "content": "[unused20]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50306": {
+      "content": "[unused21]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50307": {
+      "content": "[unused22]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50308": {
+      "content": "[unused23]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50309": {
+      "content": "[unused24]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50310": {
+      "content": "[unused25]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50311": {
+      "content": "[unused26]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50312": {
+      "content": "[unused27]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50313": {
+      "content": "[unused28]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50314": {
+      "content": "[unused29]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50315": {
+      "content": "[unused30]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50316": {
+      "content": "[unused31]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50317": {
+      "content": "[unused32]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50318": {
+      "content": "[unused33]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50319": {
+      "content": "[unused34]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50320": {
+      "content": "[unused35]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50321": {
+      "content": "[unused36]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50322": {
+      "content": "[unused37]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50323": {
+      "content": "[unused38]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50324": {
+      "content": "[unused39]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50325": {
+      "content": "[unused40]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50326": {
+      "content": "[unused41]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50327": {
+      "content": "[unused42]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50328": {
+      "content": "[unused43]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50329": {
+      "content": "[unused44]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50330": {
+      "content": "[unused45]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50331": {
+      "content": "[unused46]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50332": {
+      "content": "[unused47]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50333": {
+      "content": "[unused48]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50334": {
+      "content": "[unused49]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50335": {
+      "content": "[unused50]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50336": {
+      "content": "[unused51]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50337": {
+      "content": "[unused52]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50338": {
+      "content": "[unused53]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50339": {
+      "content": "[unused54]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50340": {
+      "content": "[unused55]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50341": {
+      "content": "[unused56]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50342": {
+      "content": "[unused57]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50343": {
+      "content": "[unused58]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50344": {
+      "content": "[unused59]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50345": {
+      "content": "[unused60]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50346": {
+      "content": "[unused61]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50347": {
+      "content": "[unused62]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50348": {
+      "content": "[unused63]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50349": {
+      "content": "[unused64]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50350": {
+      "content": "[unused65]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50351": {
+      "content": "[unused66]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50352": {
+      "content": "[unused67]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50353": {
+      "content": "[unused68]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50354": {
+      "content": "[unused69]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50355": {
+      "content": "[unused70]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50356": {
+      "content": "[unused71]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50357": {
+      "content": "[unused72]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50358": {
+      "content": "[unused73]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50359": {
+      "content": "[unused74]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50360": {
+      "content": "[unused75]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50361": {
+      "content": "[unused76]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50362": {
+      "content": "[unused77]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50363": {
+      "content": "[unused78]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50364": {
+      "content": "[unused79]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50365": {
+      "content": "[unused80]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50366": {
+      "content": "[unused81]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50367": {
+      "content": "[unused82]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50368": {
+      "content": "[Q] ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50369": {
+      "content": "[D] ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "extra_special_tokens": {},
+  "mask_token": "[MASK]",
+  "max_length": 255,
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 255,
+  "pad_to_multiple_of": null,
+  "pad_token": "[MASK]",
+  "pad_token_type_id": 0,
+  "padding_side": "right",
+  "sep_token": "[SEP]",
+  "stride": 0,
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "truncation_side": "right",
+  "truncation_strategy": "longest_first",
+  "unk_token": "[UNK]"
+}