GuillaumeSalouHF commited on
Commit
4701de6
·
unverified ·
1 Parent(s): 9f90f1f

fix(bedrock): force tool cachePoint via cache_control_injection_points

Browse files

LiteLLM 1.83.0's Bedrock Converse adapter reads cache_control blocks
on system content blocks but ignores cache_control on individual tool
dicts (only the native Anthropic adapter checks tools). Result: the
~16k tokens of tool definitions are re-billed at full input price on
every Bedrock turn instead of hitting the cache_read tier.

Pass cache_control_injection_points=[{location:tool_config}] to the
Converse adapter so it appends a cachePoint at the end of the tool
list. System-prompt caching already works via prompt_caching.py.

Expected effect: cache_read ratio on Opus 4.6 jumps from ~17% (auto-
prefix only) towards 50-70% on multi-turn sessions, dropping the
input bill ~25-35%.

Files changed (1) hide show
  1. agent/core/llm_params.py +12 -1
agent/core/llm_params.py CHANGED
@@ -160,7 +160,18 @@ def _resolve_llm_params(
160
  # (``AWS_ACCESS_KEY_ID`` / ``AWS_SECRET_ACCESS_KEY`` / ``AWS_REGION``).
161
  # The Anthropic thinking/effort shape is not forwarded through Converse
162
  # the same way, so we leave it off for now.
163
- return {"model": model_name}
 
 
 
 
 
 
 
 
 
 
 
164
 
165
  if model_name.startswith("openai/"):
166
  params = {"model": model_name}
 
160
  # (``AWS_ACCESS_KEY_ID`` / ``AWS_SECRET_ACCESS_KEY`` / ``AWS_REGION``).
161
  # The Anthropic thinking/effort shape is not forwarded through Converse
162
  # the same way, so we leave it off for now.
163
+ #
164
+ # ``cache_control_injection_points`` instructs the Converse adapter to
165
+ # append a cachePoint at the end of the tool list. Per-tool
166
+ # ``cache_control`` blocks (set by prompt_caching.py for the Anthropic
167
+ # native path) are otherwise silently dropped by Converse, leaving the
168
+ # ~16k tokens of tool defs uncached on every Bedrock turn. System-prompt
169
+ # caching still works via cache_control on system content blocks
170
+ # (Converse reads those).
171
+ return {
172
+ "model": model_name,
173
+ "cache_control_injection_points": [{"location": "tool_config"}],
174
+ }
175
 
176
  if model_name.startswith("openai/"):
177
  params = {"model": model_name}