sraivante/zip-county-city-seq2seq-21m

ZIP County City - v0.3.0 experimental, 31.49M parameters

Superseded by sraivante/zipqa-intent-150k. This model memorises the ZIP table in its weights; the successor splits the problem — a 148k-parameter parser understands the question, a deterministic engine answers from the table. The comparison:

	this model (v0.3.0)	zipqa-intent-150k
Approach	facts stored in weights	parse question → look up table row
Parameters	31,485,998	148,195 (212× fewer)
Questions it wasn't templated on	12.10% complete (10k fresh-wording test)	48/48 hand-typed eval
Question types	lookup, validate	lookup, list, count, check, nearest, top + refusal
Hinglish	no	yes
Hallucination	unknown ZIPs may produce geographic guesses	none — every fact comes from a row
Table refresh	retrain	swap the CSV

(Eval sets differ, so the accuracy rows are indicative, not a controlled benchmark — but the gap is the point.) This repo stays up as the measured record of the approach that didn't work.

Actual trained weights are included in models_s2s/model_v3.pt. The same 5,000-prompt benchmark improves from 25.74% to 68.96% complete scored responses (+43.22 percentage points; 1,287 to 3,448 correct). On a separate locked 10,000-prompt fresh-wording test, complete output improves from 2.50% to 12.10%. Intent regresses, and unfamiliar wording remains unreliable. The 68.96% result is not an estimate for arbitrary English questions or unseen ZIP codes.

The repository slug retains the original seq2seq-21m name for continuity. V3 has 31,485,998 parameters and a different architecture/output schema. It is a custom PyTorch checkpoint trained from random weights, with a learned span parser, ZIP-conditioned geographic classifiers and learned membership/validation heads. No pretrained weights, geographic CSV, database, API or retrieval are used at inference. Its English tasks are US ZIP lookup and consistency checks of supplied county, city and state. All model versions are preserved in separate versions/<version>/ folders and immutable version tags.

Load and use

pip install "torch>=2.2" "numpy>=1.26" "huggingface_hub>=0.25"

import hashlib
import json
import sys
from pathlib import Path
from huggingface_hub import hf_hub_download, snapshot_download

repo = "sraivante/zip-county-city-seq2seq-21m"
# Read the release configuration to select the exact code and weights.
config = json.loads(Path(hf_hub_download(repo, "config.json")).read_text())
release = config["release"]
root = Path(snapshot_download(repo, revision=release["revision"],
    allow_patterns=release["allow_patterns"]))
version = root / release["subfolder"]
weight_file = version / release["weights"]
assert hashlib.sha256(weight_file.read_bytes()).hexdigest() == release["weights_sha256"]
sys.path.insert(0, str(version))
from model_v3 import ZipAssistantV3
assistant = ZipAssistantV3(version / "models_s2s", device="cpu")
print(assistant.ask("zip 60601"))
print(assistant.ask("check zip 60601, city CHICAGO, county COOK, state IL"))

These measured CPU examples return ZIP 60601, IL, COOK, CHICAGO; the second is classified as validation with valid: true. ask_many([...]) supports batches. Vocabularies are embedded in the checkpoint. Inference runs offline after download. CPU loading was tested with Python 3.13 and PyTorch 2.12.0+cpu; minimum dependency versions were not individually tested. This is not a Transformers AutoModel.

Output: query, intent (lookup/validate), ZIP string (leading zeros preserved), state, representative county/city, given_copied, given_resolved (both are source copies, not spelling corrections), valid, uncalibrated verdict_confidence, and four character-span pairs. Confidence is the model probability of validity, not calibrated certainty that the chosen answer is correct. Inputs exceeding 160 characters raise an error. Unknown ZIPs may still produce geographic guesses.

Download statistics

The loading example reads config.json to select the immutable release revision, code paths and checkpoint SHA-256. Hugging Face counts requests to this standard query file in the repository's aggregate download statistics. See Hugging Face's counting rules.

These are aggregate download counts, not a list of people who downloaded the model. Downloading only a .pt file or following an old tagged example that skips config.json is not covered by this default counter. Offline inference does not contact the Hub. Previous releases, their files and version tags are preserved.

Measured improvement and regressions

Metric	v0.2.0	v0.3.0
Complete scored response	25.74%	68.96%
ZIP	87.56%	99.98%
State	88.58%	99.98%
Any listed county	87.80%	99.98%
Any listed city	42.32%	99.98%
Intent	91.22%	90.16% (regression)
Validation verdict	51.47%	71.71%

The same 5,000 prompts and labels were used; verdict uses only the 3,750 validation prompts (1,930 to 2,689 correct). Complete scored response checks intent, ZIP, state, independently acceptable county/city, and verdict where applicable. It does not score supplied-name extraction or exact county/city pair consistency. The original v0.1.0 result was 14.22%. These are known ZIP facts, and the old benchmarks informed development; they are regression tests, not untouched tests.

Separate evaluation	v0.2.0	v0.3.0
Expanded 59,952 prompts, complete common output	6.61%	46.65%
Expanded intent	94.29%	78.65% (regression)
Fresh 10,000 prompts, complete common output	2.50%	12.10%
Fresh geography	43.69%	97.77%
Fresh intent	80.55%	80.00% (regression)
Fresh validation verdict (8,000 prompts)	54.84%	57.125%

Complete common output (all_13) uses the old 13-slot conditions with fixed EOS; v3 emits separate fields. Fresh all_fields additionally checks supplied state, which v2 has no output slot for (v2 1.00%, v3 12.10%). Supplied-name component scores use bare names in v3 versus state-qualified tokens in v2. Spelling tests are retained despite v3 having no spelling-correction mechanism. Full definitions, per-task/per-template scores and paired fixes/regressions are in evaluation/RESULTS.md, evaluation/result_analysis.json and the linked dataset's saved predictions. All 84,952 saved prediction checks were independently recomputed locally, including the v2 fresh-test baseline.

Architecture, training and selection

Three-layer character Transformer, width 128, four attention heads, feed-forward width 384, dropout 0.1, learned start/end pointers for ZIP/county/city/state plus intent. A learned 100,001 x 128 ZIP embedding drives 59 state, 1,936 county and 35,925 city classifiers. Alias embeddings use width 96, per-field compatibility, and a learned 6 -> 32 -> 1 validity head. Numeric ZIP indexing selects a learned floating-point parameter row; it does not retrieve source records.

Run internal_v3_20261021_01, 2026-10-21, NVIDIA L4, PyTorch 2.11.0+cu130, CUDA 13.0, BF16; seed 20261021. All 6,000 planned updates completed in 252.56 seconds within the 45-minute training cap. Parser batch 256; geography and each alias batch 1,024. AdamW betas (0.9, 0.98), epsilon 1e-8; parser LR 0.0008 and weight decay 0.01; memory LR 0.01 and weight decay 0; 150-step warmup, cosine floor 25%, gradient clip 5. The trainer records every loss and all settings. Total parser presentations: 1,536,000; geography and each alias stream: 6,144,000.

Selected step 5,500, by 4,000 development examples evaluated every 500 steps: 24.925% complete response versus 23.425% at step 6,000. ZIP/state >=98.5% gates take priority. Tests were not used for selection. The exported weights are the selected checkpoint, not the final optimizer state. Architecture, capacity and data changed together; this trial does not isolate the cause of each gain.

Exact data and reproducibility

Paired dataset release, pinned revision 21de9dd9ad1a5fcd4490f14469513a96fc59d473. It includes the exact archived training input, materialized 360,000 train / 4,000 development / 10,000 test prompts, 41,089 canonical ZIP facts, membership targets (254,178 county / 334,926 city / 205,445 state), JSONL/NPZ, source CSV, templates, vocabularies, regression suites, generation/training code and predictions. No separate fine-tuning stage or pretrained base exists. The source has 98,597 rows and 59 state/territory labels. It is the same owner-supplied MySQL snapshot as v0.2.0, not a new geographic revision. Entire template strings and normalized texts are disjoint across fresh splits; all ZIP facts remain training knowledge.

Input archive SHA-256: b18ce0f286808410bb40fe66821685be605e9c01ae704257a7ce42fedbb14f5b. Data manifest SHA-256: f3b89fdc90fd2133e443991315413d3ce25c3a24670be8cfd4b88bfc3f82137c. Source CSV SHA-256: 9ca02a431e2af1429f9cf64942ef741a3dd5223a8e8b1e635ed7c371c86ccf9b. Weights SHA-256: 8b36d48fd06f34048238756cfe98dc1e7248dd0c71452119916eb57acaf33e5e. Per-file hashes and source-code hashes are in data_provenance.json and release_manifest.json. No Git source revision was recorded; file hashes identify actual code. No training data was reconstructed or edited after the run. Historical cards retain their preparation-time status; this release records completed training and authorized publication.

To reproduce exact evaluation, extract the dataset's archive/training_input.zip in a separate directory, add the v3 weight as models_s2s/model_v3.pt, and run:

python evaluate_v3.py --model models_s2s --data data/model_v3 --out evaluation_replay --device cpu

The archive contains the original v2 baseline wrapper/weights for the separate baseline evaluation. Data is used to score outputs; v3 inference uses only its checkpoint. Training reproduction uses the archived train_v3.py configuration; optimizer/RNG resume state is excluded from this inference release.

Observed limitations and diagnosis

All 2,000 fresh lookup prompts were misclassified as validation, with spurious supplied-name spans. Seven of ten fresh template strings scored zero complete responses. A correctly extracted ZIP 60601 / CHICAGO / COOK / wrong state CA was incorrectly accepted. Overfitting to wording and false accepts remain material. Feeding gold parsed fields to unchanged validation heads gave 92.5125% (7,401/8,000) in a post-test CPU diagnostic, versus 57.125% end to end. This is not an achieved end-to-end score and was not used to choose the checkpoint; 599 false accepts remained even with gold parsing.

Next experiments should broaden wording and missing-field training, add a learned field-presence objective, cover wrong-state alternatives and harder county/city negatives, and gate intent/validation on development data. These follow-up changes have not been trained in this release. Future tuning needs a new locked test. Arbitrary typos, full state names, multiple ZIPs, unknown geography, non-English text, and unsupported inputs can fail. There is no reliable abstention. Accepted aliases are checked independently by field; this is not authoritative postal validation or street-address deliverability. Representative names follow the last primary-flagged source row, else the first; this is not verified USPS preference.

License, ownership and provenance

Copyright (c) 2026 sraivante for original code, templates, annotations and original selection/arrangement, under Apache License 2.0. Third-party ownership and terms remain separate. The owner-supplied geographic export's upstream provider, revision, collection date and license are unrecorded; no verified Apache license over all upstream material is claimed. No source is identified as nonredistributable in supplied documentation. See NOTICE. No base-model license applies because weights were initialized randomly.

All preserved releases: VERSIONS.md.