branch: elpa/gptel
commit bfc134bb94eea89350e2cdb1c1896527da99ca49
Author: Andrea Arcangeli <[email protected]>
Commit: GitHub <[email protected]>
gptel-context: Move the context before the prompt (#1110)
gptel-context.el (gptel-context--wrap-in-buffer): Place the
context chunks before the system message instead of after.
The objective of this change is to enhance KV cache reuse, based
on the following empirical observations:
1. Context chunks are large; the rest of the system prompt is
small (on average).
2. The instructions change more often than the context does in a
conversation (or rewrite/refactoring task).
So we want the move the smaller, more frequently changing
component further down in the chain.
Observation by Andrea Arcangeli (@aagit):
"As a bonus, the model seem to follow the prompt better when it's
at the end (at least open weight qwen3 coder instruct), for
example it starts to follow the instruction not add markdown
fences around the rewritten code that is otherwise ignored on a
long context.
Supposedly the fact the context format is actively using the
markdown quotes, washes the directive not to use the markdown
quotes if it's done at the top of the system message."
-----
This change is experimental, and if we see regressions in LLMs'
tendency to follow instructions, it may be reverted.
---
gptel-context.el | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/gptel-context.el b/gptel-context.el
index fb47a4cd8f2..8f73a185860 100644
--- a/gptel-context.el
+++ b/gptel-context.el
@@ -387,14 +387,14 @@ This modifies the buffer."
(cl-etypecase gptel--system-message
(string
(setq gptel--system-message
- (concat gptel--system-message "\n\n" context-string)))
+ (concat context-string "\n\n" gptel--system-message)))
(function
(setq gptel--system-message
(gptel--parse-directive gptel--system-message 'raw))
(gptel-context--wrap-in-buffer context-string))
(list
(setq gptel--system-message ;cons a new list to avoid mutation
- (cons (concat (car gptel--system-message) "\n\n"
context-string)
+ (cons (concat context-string "\n\n" (car
gptel--system-message))
(cdr gptel--system-message)))))
(setq gptel--system-message context-string))))
('user
