branch: elpa/gptel
commit 0e991255f6407d79b9d8aac41d63bd0a75d017c3
Author: Karthik Chikmagalur <[email protected]>
Commit: Karthik Chikmagalur <[email protected]>
gptel-context: Add uniform spec for targeted context
Expand context specification to support partial buffer and file
inclusion. Users can now specify specific regions (by position
or line ranges) instead of only full buffer/file text. This
enables more precise context control for LLM requests.
See the documentation of `gptel-context' for more information on
the specification.
The most important change is that portions of buffers can be
specified using position or line ranges, and not just overlays.
This make context state handling for ephemeral uses (like presets)
very simple.
* gptel-context.el: Change context storage from (buf . overlays)
to (source . spec) format to support :overlays, :lines, and
:bounds keys in the plist spec.
(gptel-context--make-overlay): Update to use plist storage.
(gptel-context--collect): Modify to handle new plist format and
clean dead overlays in place.
(gptel-context--insert-buffer-string): Refactor to accept
context-data plist instead of overlays list, merge all region
sources (overlays/bounds/lines) into unified list, and support
optional header parameter.
(gptel-context--insert-file-string): New function to handle
partial file inclusion using :lines and :bounds specifications.
NOTE: Currently reading in parts of a file requires reading in the
whole file, unless the file is already being visited. Seeking to
points or line-ends directly is not implemented yet.
(gptel-context--string): Update to use new (source . spec) format.
* gptel-request.el (gptel-context): Expand gptel-context docstring
to document new targeted context specification format with
:bounds, :lines, :overlays, and :mime keys for both buffers and
files.
* README.org (Additional Configuration):
Add gptel-context to request settings table. Update preset
documentation to show :context usage with file list example.
---
README.org | 8 ++--
gptel-context.el | 115 +++++++++++++++++++++++++++++++++++++++++--------------
gptel-request.el | 32 ++++++++++------
3 files changed, 112 insertions(+), 43 deletions(-)
diff --git a/README.org b/README.org
index 4638b9bdf7b..e82029e1157 100644
--- a/README.org
+++ b/README.org
@@ -1750,6 +1750,7 @@ Other Emacs clients for LLMs prescribe the format of the
interaction (a comint s
| =gptel-temperature= | Randomness in response text, 0 to 2.
|
| =gptel-cache= | Cache prompts, system message or tools (Anthropic
only) |
| =gptel-use-context= | How/whether to include additional context
|
+| =gptel-context= | List of context sources (files/buffers) for queries
|
| =gptel-use-tools= | Disable, allow or force LLM tool-use
|
| =gptel-tools= | List of tools to include with requests
|
|-----------------------+---------------------------------------------------------|
@@ -1820,7 +1821,7 @@ More generally, you can specify a bundle of options:
:tools '("read_buffer" "modify_buffer")) ;gptel tools or tool names
#+end_src
-Besides a couple of special keys (=:description=, =:parents= to inherit other
presets), there is no predefined list of keys. Instead, the key =:foo=
corresponds to setting =gptel-foo= (preferred) or =gptel--foo=. So the preset
can include the value of any gptel option. For example, the following preset
sets =gptel-temperature= and =gptel-use-context=:
+Besides a couple of special keys (=:description=, =:parents= to inherit other
presets), there is no predefined list of keys. Instead, the key =:foo=
corresponds to setting =gptel-foo= (preferred) or =gptel--foo=. So the preset
can include the value of any gptel option. For example, the following preset
sets =gptel-temperature=, =gptel-use-context= and =gptel-context=, a list of
files to include as context:
#+begin_src emacs-lisp
(gptel-make-preset 'proofreader
@@ -1828,8 +1829,9 @@ Besides a couple of special keys (=:description=,
=:parents= to inherit other pr
:backend "ChatGPT"
:model 'gpt-4.1-mini
:tools '("read_buffer" "spell_check" "grammar_check")
- :temperature 0.7 ;sets gptel-temperature
- :use-context 'system) ;sets gptel-use-context
+ :use-context 'system ;sets gptel-use-context
+ :context '("./.grammar_rules.md" "./jargonfile.md") ;sets gptel-context
+ :temperature 0.2) ;sets gptel-temperature
#+end_src
Switching to a preset applies the specified settings without affecting other
settings. Depending on the scope option (~=~ in gptel's transient menu),
presets can be applied globally, buffer-locally or for the next request only.
diff --git a/gptel-context.el b/gptel-context.el
index 4766515bf49..ca460e4b45c 100644
--- a/gptel-context.el
+++ b/gptel-context.el
@@ -346,12 +346,14 @@ afterwards."
"Highlight the region from START to END.
ADVANCE controls the overlay boundary behavior."
- (let ((overlay (make-overlay start end nil (not advance) advance)))
+ (let ((overlay (make-overlay start end nil (not advance) advance))
+ (buf-entry (alist-get (current-buffer) gptel-context)))
(overlay-put overlay 'evaporate t)
(overlay-put overlay 'face 'gptel-context-highlight-face)
(overlay-put overlay 'gptel-context t)
- (push overlay (alist-get (current-buffer)
- gptel-context))
+ (setf (alist-get (current-buffer) gptel-context)
+ (plist-put buf-entry :overlays
+ (cons overlay (plist-get buf-entry :overlays))))
overlay))
;;;###autoload
@@ -452,15 +454,16 @@ Ignore overlays, buffers and files that are not live or
readable."
(let ((res))
(dolist (entry (or context-alist gptel-context))
(pcase entry ;Context entry is:
- (`(,buf . ,ovs)
+ (`(,buf . ,data)
(cond
- ((buffer-live-p buf) ;Overlay(s) in a buffer
- (if-let* ((live-ovs (cl-loop for ov in ovs
- when (overlay-start ov)
- collect ov)))
- (push (cons buf live-ovs) res)))
+ ((buffer-live-p buf)
+ ;; (<buf> :overlays ... :lines ... :bounds ...)
+ (when-let* ((ovs (plist-get data :overlays))) ;Clear dead overlays
+ (plist-put data :overlays (cl-remove-if-not #'overlay-start ovs)))
+ (push (cons buf data) res))
((and (stringp buf) (file-readable-p buf))
- (push (cons buf ovs) res)))) ;A file list with (maybe) a mimetype
+ ;; ("/file/path" :mime ... :bounds ... :line ...)
+ (push (cons buf data) res))))
((and (pred stringp) (pred file-readable-p)) ;Just a file, figure out
mimetype
(if (file-directory-p entry)
@@ -482,21 +485,56 @@ Ignore overlays, buffers and files that are not live or
readable."
((pred buffer-live-p) (push (list entry) res)))) ;Just a buffer
res))
-(defun gptel-context--insert-buffer-string (buffer overlays)
- "Insert at point a context string from all OVERLAYS in BUFFER.
+(defun gptel-context--insert-buffer-string (buffer context-data &optional
header)
+ "Insert at point a context string from CONTEXT-DATA in BUFFER.
-If OVERLAYS is nil add the entire buffer text."
+CONTEXT-DATA is a plist with keys :overlays, :lines and :bounds to
+include specific overlays, line ranges or position bounds instead of the
+entire buffer. See `gptel-context'.
+
+HEADER is an optional header to insert before the contents."
(let ((is-top-snippet t)
- (previous-line 1))
- (insert (format "In buffer `%s`:" (buffer-name buffer))
- "\n\n```" (gptel--strip-mode-suffix (buffer-local-value
- 'major-mode buffer))
+ (previous-line 1)
+ regions)
+ ;; Collect all regions into a unified list of (start . end) pairs
+ (with-current-buffer buffer
+ (without-restriction
+ ;; Collect overlays
+ (dolist (ov (plist-get context-data :overlays))
+ (when (overlay-start ov)
+ (push (cons (overlay-start ov) (overlay-end ov))
+ regions)))
+ ;; Collect bounds (already in position format)
+ (when-let* ((bounds (plist-get context-data :bounds)))
+ (if (consp (car bounds))
+ (nconc regions bounds) ;((start1 . end1) (start2 . end2) ...)
+ (push bounds regions))) ;(start1 . end1)
+ ;; Collect lines (convert line numbers to positions)
+ (when-let* ((line-bounds (plist-get context-data :lines)))
+ ;; Convert singleton (start1 . end1) to ((start1 . end1))
+ (unless (consp (car line-bounds)) (setq line-bounds (list
line-bounds)))
+ (dolist (pair line-bounds)
+ (push (cons (progn (goto-char (point-min))
+ (forward-line (1- (car pair)))
+ (point))
+ (progn (goto-char (point-min))
+ (forward-line (cdr pair))
+ (point)))
+ regions)))))
+
+ ;; TODO: Update sort for Emacs 28+ calling convention
+ ;; Sort by start position. In-place, but assign to be sure.
+ (setq regions (sort regions (lambda (a b) (< (car a) (car b)))))
+
+ ;; Insert header
+ (insert (or header (format "In buffer `%s`:\n\n```"(buffer-name buffer)))
+ (gptel--strip-mode-suffix (buffer-local-value
+ 'major-mode buffer))
"\n")
- (if (not overlays)
+ (if (not regions)
(insert-buffer-substring-no-properties buffer)
- (dolist (context overlays)
- (let* ((start (overlay-start context))
- (end (overlay-end context)))
+ (dolist (region regions)
+ (let ((start (car region)) (end (cdr region)))
(let (lineno column)
(with-current-buffer buffer
(without-restriction
@@ -515,10 +553,31 @@ If OVERLAYS is nil add the entire buffer text."
(setq is-top-snippet nil)
(unless (= previous-line lineno) (insert "\n"))))
(insert-buffer-substring-no-properties buffer start end)))
- (unless (>= (overlay-end (car (last overlays))) (point-max))
+ (unless (>= (cdr (car (last regions))) (point-max))
(insert "\n...")))
(insert "\n```")))
+(defun gptel-context--insert-file-string (path &optional spec)
+ "Insert at point the contents of file at PATH as context.
+
+SPEC is a plist specifying :lines or position :bounds to include instead
+of the entire file. See `gptel-context' for details."
+ (if (not (and spec (or (plist-member spec :lines)
+ (plist-member spec :bounds))))
+ ;; Insert whole file
+ (gptel--insert-file-string path)
+ ;; Insert only regions from lines and/or bounds
+ (let* ((visiting-buf (find-buffer-visiting ;Reuse buffer
+ path (lambda (b) (not (buffer-modified-p b)))))
+ (file-buf (or visiting-buf ;temp buf to dump file contents
+ (gptel--temp-buffer " *gptel-file-context*"))))
+ (unless visiting-buf
+ (with-current-buffer file-buf (insert-file-contents path)))
+ (gptel-context--insert-buffer-string
+ file-buf spec (format "In file `%s`:\n\n```\n"
+ (abbreviate-file-name path)))
+ (unless visiting-buf (kill-buffer file-buf)))))
+
(defun gptel-context--string (context-alist)
"Format the aggregated gptel context as annotated markdown fragments.
@@ -526,12 +585,12 @@ Returns a string. CONTEXT-ALIST is a structure containing
context overlays, see `gptel-context'."
(with-temp-buffer
(cl-loop for entry in context-alist
- for (buf . ovs) = (ensure-list entry)
- if (bufferp buf)
- do (gptel-context--insert-buffer-string buf ovs)
- else if (or (not (plist-get ovs :mime))
- (string-match-p "^text/" (plist-get ovs :mime)))
- do (gptel--insert-file-string buf) end
+ for (source . spec) = (ensure-list entry)
+ if (bufferp source)
+ do (gptel-context--insert-buffer-string source spec)
+ else if (or (not (plist-get spec :mime))
+ (string-match-p "^text/" (plist-get spec :mime)))
+ do (gptel-context--insert-file-string source spec) end
do (insert "\n\n")
finally do
(skip-chars-backward "\n\t\r ")
diff --git a/gptel-request.el b/gptel-request.el
index 6c66cf988f3..3d059e05306 100644
--- a/gptel-request.el
+++ b/gptel-request.el
@@ -703,28 +703,36 @@ name):
#<buffer *scratch*>
...)
-You can also specify context sources with more detail. Overlay regions
-in buffers can be specified as
+The above covers the most common cases. You can also specify context
+sources in a more targeted way, with entries of the form
- (buf ov1 ov2 ...)
+ (<buffer> . spec)
+ (\"/path/to/file\" . spec)
-where ov1, ov2 are overlays. In this case the text of the overlay
-regions is sent instead of the text of the entire buffer.
+where spec is a plist declaring specific parts of the buffer/file to
+include instead of the entire text.
-Instead of as a string, file paths can also be specified along with
-their MIME-types:
+For buffers, you can specify regions to include using buffer spans and
+line number ranges as conses, and overlays as a list:
- (\"/path/to/image\" :mime \"image/png\")
+ (<buffer> :bounds ((start1 . end1) (start2 . end2) ...)
+ :lines ((from1 . to1) (from2 . end2) ...)
+ :overlays (ov1 ov2 ...))
+
+For files, spec can include buffer spans and line number ranges, as well as
+the MIME type of the file:
+
+ (\"/path/to/file\" :bounds ((start1 . end1) (start2 . end2) ...)
+ :lines ((from1 . to1) (from2 . end2) ...)
+ :mime \"image/png\")
gptel tries to guess file MIME types, but is not always successful, so
-it is recommended to provide it with non-text files. Additional plist
-keys (besides :mime) are ignored, but support for more keys may be
-implemented in the future.
+it is recommended to provide it with non-text files.
Usage of context commands (such as `gptel-add' and `gptel-add-file')
will modify this variable. You can also set this variable
buffer-locally, or let-bind it around calls to gptel queries, or via
-gptel presets."
+gptel presets with the :context key."
:type '(repeat string))
(defcustom gptel-markdown-validate-link #'always
