Spaces:
Running on CPU Upgrade
Running on CPU Upgrade
GuillaumeSalouHF commited on
fix(bedrock): force tool cachePoint via cache_control_injection_points
Browse filesLiteLLM 1.83.0's Bedrock Converse adapter reads cache_control blocks
on system content blocks but ignores cache_control on individual tool
dicts (only the native Anthropic adapter checks tools). Result: the
~16k tokens of tool definitions are re-billed at full input price on
every Bedrock turn instead of hitting the cache_read tier.
Pass cache_control_injection_points=[{location:tool_config}] to the
Converse adapter so it appends a cachePoint at the end of the tool
list. System-prompt caching already works via prompt_caching.py.
Expected effect: cache_read ratio on Opus 4.6 jumps from ~17% (auto-
prefix only) towards 50-70% on multi-turn sessions, dropping the
input bill ~25-35%.
- agent/core/llm_params.py +12 -1
agent/core/llm_params.py
CHANGED
|
@@ -160,7 +160,18 @@ def _resolve_llm_params(
|
|
| 160 |
# (``AWS_ACCESS_KEY_ID`` / ``AWS_SECRET_ACCESS_KEY`` / ``AWS_REGION``).
|
| 161 |
# The Anthropic thinking/effort shape is not forwarded through Converse
|
| 162 |
# the same way, so we leave it off for now.
|
| 163 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 164 |
|
| 165 |
if model_name.startswith("openai/"):
|
| 166 |
params = {"model": model_name}
|
|
|
|
| 160 |
# (``AWS_ACCESS_KEY_ID`` / ``AWS_SECRET_ACCESS_KEY`` / ``AWS_REGION``).
|
| 161 |
# The Anthropic thinking/effort shape is not forwarded through Converse
|
| 162 |
# the same way, so we leave it off for now.
|
| 163 |
+
#
|
| 164 |
+
# ``cache_control_injection_points`` instructs the Converse adapter to
|
| 165 |
+
# append a cachePoint at the end of the tool list. Per-tool
|
| 166 |
+
# ``cache_control`` blocks (set by prompt_caching.py for the Anthropic
|
| 167 |
+
# native path) are otherwise silently dropped by Converse, leaving the
|
| 168 |
+
# ~16k tokens of tool defs uncached on every Bedrock turn. System-prompt
|
| 169 |
+
# caching still works via cache_control on system content blocks
|
| 170 |
+
# (Converse reads those).
|
| 171 |
+
return {
|
| 172 |
+
"model": model_name,
|
| 173 |
+
"cache_control_injection_points": [{"location": "tool_config"}],
|
| 174 |
+
}
|
| 175 |
|
| 176 |
if model_name.startswith("openai/"):
|
| 177 |
params = {"model": model_name}
|