bug#72945: 29.4; Org: ox-html: attr_html not supported in source code and fixed-width blocks during HTML export

[Suhail Singh]

Since more than a month has gone by, I am creating this issue to ease
tracking.  This issue was first mentioned in this post:
.  Details
cross-posted below.

Unlike example blocks, source code and fixed-width blocks don't support
the attr_html keyword.  Contrast these outputs:

#+begin_src emacs-lisp :results value replace :wrap src html
  (require 'org)
  (require 'ox-html)

  (org-export-string-as
   "#+attr_html: :class foo
  ,#+begin_src sh :exports code
pwd
  ,#+end_src"
   'html t)
#+end_src

#+RESULTS:
#+begin_src html

pwd


#+end_src


#+begin_src emacs-lisp :results value replace :wrap src html
  (require 'org)
  (require 'ox-html)

  (org-export-string-as
   "#+attr_html: :class foo
  ,#+RESULTS:
  : blah"
   'html t)
#+end_src

#+RESULTS:
#+begin_src html

blah

#+end_src


With the output for example blocks:

#+begin_src emacs-lisp :results value replace :wrap src html
  (require 'org)
  (require 'ox-html)

  (org-export-string-as
   "#+attr_html: :class foo
  ,#+begin_example
hello world!
  ,#+end_example"
   'html t)
#+end_src

#+RESULTS:
#+begin_src html

hello world!

#+end_src



In GNU Emacs 29.4 (build 2, x86_64-suse-linux-gnu, GTK+ Version 3.24.43,
cairo version 1.18.0)
System Description: openSUSE Tumbleweed

Configured using:
 'configure --disable-build-details --without-pop --with-mailutils
 --with-native-compilation --without-hesiod --with-gameuser=:games
 --with-kerberos --with-kerberos5 --with-file-notification=inotify
 --with-modules --enable-autodepend --prefix=/usr
 --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share
 --localstatedir=/var --sharedstatedir=/var/lib
 --libexecdir=/usr/libexec --with-file-notification=yes
 --libdir=/usr/lib64
 
--enable-locallisppath=/usr/share/emacs/29.4/site-lisp:/usr/share/emacs/site-lisp
 --with-x --with-xim --with-sound --with-xpm --with-jpeg --with-tiff
 --with-gif --with-png --with-rsvg --with-dbus --with-webp --with-xft
 --with-imagemagick --without-gpm --with-x-toolkit=gtk3 --with-pgtk
 --with-toolkit-scroll-bars --x-includes=/usr/include
 --x-libraries=/usr/lib64 --with-libotf --with-m17n-flt --with-cairo
 --with-xwidgets --build=x86_64-suse-linux --with-dumping=pdumper
 'CFLAGS=-O2 -Wall -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3
 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables
 -fstack-clash-protection -Werror=return-type -flto=auto -g
 -D_GNU_SOURCE -DGDK_DISABLE_DEPRECATION_WARNINGS
 -DGLIB_DISABLE_DEPRECATION_WARNINGS -pipe -Wno-pointer-sign
 -Wno-unused-variable -Wno-unused-label -fno-optimize-sibling-calls
 -DPDMP_BASE='\''"emacs-wayland"'\''' LDFLAGS=-Wl,-O2'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ
IMAGEMAGICK JPEG JSON LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 MODULES
NATIVE_COMP NOTIFY INOTIFY PDUMPER PGTK PNG RSVG SECCOMP SOUND SQLITE3
THREADS TIFF TOOLKIT_SCROLL_BARS TREE_SITTER WEBP XIM XWIDGETS GTK3 ZLIB

Important settings:
  value of $LC_NUMERIC: POSIX
  value of $LC_TIME: en_US.UTF-8
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Fundamental

Minor modes in effect:
  eros-mode: t
  yas-global-mode: t
  yas-minor-mode: t
  global-orglink-mode: t
  TeX-PDF-mode: t
  abridge-diff-mode: t
  global-git-commit-mode: t
  magit-auto-revert-mode: t
  envrc-global-mode: t
  envrc-mode: t
  which-key-mode: t
  flycheck-status-emoji-mode: t
  global-flycheck-mode: t
  coterm-mode: t
  shell-dirtrack-mode: t
  selected-global-mode: t
  selected-minor-mode: t
  async-bytecomp-package-mode: t
  marginalia-mode: t
  pixel-scroll-precision-mode: t
  which-function-mode: t
  save-place-mode: t
  repeat-mode: t
  minibuffer-depth-indicate-mode: t
  electric-pair-mode: t
  my/keys-minor-mode: t
  override-global-mode: t
  el-patch-use-package-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  blink-cursor-mode: t
  buffer-read-only: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  view-mode: t

Load-path shadows:
/home/user/.emacs.d/elpa/helm-4.0/helm-packages hides 
/home/user/.emacs.d/elpa/helm-core-4.0/helm-packages
/home/user/.emacs.d/elpa/magit-4.0.0/magit hides 
/home/user/.emacs.d/elpa/git-commit-4.1.0/magit
/home/user/.emacs.d/elpa/magit-4.0.0/magit-worktree hides 
/home/user/.emacs.d/elpa/git-commit-4.1.0/magit-worktree
/home/user/.emacs.d/elpa/magit-4.0.0/magit-wip hides 
/home/user/.emacs.d/elpa/git-commit-4.1.0/magit-wip
/home/user/.emacs.d/elpa/magit-4.0.0/magit-transient hides 
/home/user/.emacs.d/elpa/git-commit-4.1.0/magit-transient
/home/user/.emacs.d/elpa/magit-4.0.0/magit-tag hides 
/home/user/.emacs.d/elpa/git-commit-4.1.0/magit-tag
/home/user/.emacs.d/elpa/magit-4.0.0/magit-subtree hides 
/home/u

bug#72945: [PATCH 2/2] ox-html: Add support for attr_html in fixed-width blocks

[Suhail Singh]

* lisp/ox-html.el (org-html-fixed-width): Handle attr_html in a manner
similar to example blocks.

TINYCHANGE
---
 lisp/ox-html.el | 20 +++-
 1 file changed, 15 insertions(+), 5 deletions(-)

diff --git a/lisp/ox-html.el b/lisp/ox-html.el
index 3e83f02e2..a85b1e6ac 100644
--- a/lisp/ox-html.el
+++ b/lisp/ox-html.el
@@ -2728,13 +2728,23 @@ (defun org-html-export-block
 
  Fixed Width
 
-(defun org-html-fixed-width (fixed-width _contents _info)
+(defun org-html-fixed-width (fixed-width _contents info)
   "Transcode a FIXED-WIDTH element from Org to HTML.
 CONTENTS is nil.  INFO is a plist holding contextual information."
-  (format "\n%s"
- (org-html-do-format-code
-  (org-remove-indentation
-   (org-element-property :value fixed-width)
+  (let ((attributes (org-export-read-attribute :attr_html fixed-width)))
+(if-let ((class-val (plist-get attributes :class)))
+(setq attributes (plist-put attributes :class (concat "example " 
class-val)))
+  (setq attributes (plist-put attributes :class "example")))
+(format "\n%s"
+(let* ((reference (org-html--reference fixed-width info))
+  (a (org-html--make-attribute-string
+  (if (or (not reference) (plist-member attributes :id))
+  attributes
+(plist-put attributes :id reference)
+ (if (org-string-nw-p a) (concat " " a) ""))
+   (org-html-do-format-code
+(org-remove-indentation
+ (org-element-property :value fixed-width))
 
  Footnote Reference
 
-- 
2.46.0

bug#72945: [PATCH 1/2] ox-html: Add support for attr_html for source code blocks

[Suhail Singh]

* lisp/ox-html.el (org-html-src-block): Add org-src-container class to
any class attributes specified via the attr_html keyword.

TINYCHANGE
---
 lisp/ox-html.el | 14 --
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/lisp/ox-html.el b/lisp/ox-html.el
index 446698758..3e83f02e2 100644
--- a/lisp/ox-html.el
+++ b/lisp/ox-html.el
@@ -3667,14 +3667,24 @@ (defun org-html-src-block
 contextual information."
   (if (org-export-read-attribute :attr_html src-block :textarea)
   (org-html--textarea-block src-block)
-(let* ((lang (org-element-property :language src-block))
+(let* ((attributes (org-export-read-attribute :attr_html src-block))
+   (lang (org-element-property :language src-block))
   (code (org-html-format-code src-block info))
   (label (let ((lbl (org-html--reference src-block info t)))
(if lbl (format " id=\"%s\"" lbl) "")))
   (klipsify  (and  (plist-get info :html-klipsify-src)
 (member lang '("javascript" "js"
   "ruby" "scheme" "clojure" "php" 
"html")
-  (format "\n%s%s\n"
+  (if-let ((class-val (plist-get attributes :class)))
+  (setq attributes (plist-put attributes :class (concat 
"org-src-container " class-val)))
+(setq attributes (plist-put attributes :class "org-src-container")))
+  (format "\n%s%s\n"
+  (let* ((reference (org-html--reference src-block info))
+(a (org-html--make-attribute-string
+(if (or (not reference) (plist-member attributes :id))
+attributes
+  (plist-put attributes :id reference)
+   (if (org-string-nw-p a) (concat " " a) ""))
  ;; Build caption.
  (let ((caption (org-export-get-caption src-block)))
(if (not caption) ""

base-commit: 3428b522fb83c793d0915d63a30601eb993e2b30
-- 
2.46.0

bug#72945: 29.4; Org: ox-html: attr_html not supported in source code and fixed-width blocks during HTML export

[Suhail Singh]

Eli Zaretskii  writes:

> Thanks, but why are you posting this here?  ox-html.el is part of Org,

My reasoning was that Org is a part of Emacs, so debbugs would be
appropriate.  Or at least, that it would _not be inappropriate_.
Perhaps I was mistaken.

> and I don't see anything in the above discussion that indicates you
> should post these patches to the Emacs bug-tracker.  What did I miss?

There was no normative expectation to report these to debbugs, i.e., I
am doing so of my own volition.  Reading between the lines of your
response, however, it seems debbugs may not be the right place for it.
If so, apologies for the noise; especially if this was documented
somewhere and I missed it.

My intent was to have a reliable reference that I could either subscribe
to or check semi-regularly, to understand the status of the issue.
Specifically, if/when the issue gets resolved, I'd like to know without
having to monitor all the commits in the Org repository.

I understand that Org uses some custom mailing list software ("Woof!")
to help with tracking issues.  However, it's not clear whether that is
exclusively the case (i.e., debbugs is not to be used for Org issues).
On a related note, "Woof!" doesn't list the issue under "Bugs" and I
don't quite understand why:
.

In any case, thank you for taking the time to look into this.

-- 
Suhail

bug#73046: 29.4; Emacs 100% CPU usage for several seconds when opening dired buffer over TRAMP

[Suhail Singh]

I observe 100% CPU usage for several seconds when invoking dired for a
directory on one of the remote machines I connect to.  This doesn't seem
to happen for another host.  The connection to the affected host is
slower, and on that host the issue is worse when the directory has
several symlinks.

To reproduce via =emacs -Q=:
- Invoke `find-file' and connect to problematic host over ssh.
- Once connection is established, open dired pointing to a directory on
  the affected host.
- Observe bug: it takes _several_ seconds for the dired buffer to show.
  During that time, Emacs CPU usage is 100%.
- Once dired buffer is open, invoke `revert-buffer' and observe issue
  again.

Disabling global-font-lock-mode, makes the situation better.

Given that rudimentary font-locking should be possible via, simply, the
output of `ls --dired', it is unclear why enabling font-locking makes
things so much worse.  If there are some aspects of font-locking that
perform much worse over a slower connection, it would help if there were
a user configuration to disable them.

It would also help if being over a slow connection didn't result in
Emacs consuming 100% of the CPU via functions such as
`tramp-wait-for-regexp' (based on profiler-report).  Could some of this
be done asynchronously?


In GNU Emacs 29.4 (build 2, x86_64-suse-linux-gnu, GTK+ Version 3.24.43,
cairo version 1.18.0)
System Description: openSUSE Tumbleweed

Configured using:
 'configure --disable-build-details --without-pop --with-mailutils
 --with-native-compilation --without-hesiod --with-gameuser=:games
 --with-kerberos --with-kerberos5 --with-file-notification=inotify
 --with-modules --enable-autodepend --prefix=/usr
 --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share
 --localstatedir=/var --sharedstatedir=/var/lib
 --libexecdir=/usr/libexec --with-file-notification=yes
 --libdir=/usr/lib64
 
--enable-locallisppath=/usr/share/emacs/29.4/site-lisp:/usr/share/emacs/site-lisp
 --with-x --with-xim --with-sound --with-xpm --with-jpeg --with-tiff
 --with-gif --with-png --with-rsvg --with-dbus --with-webp --with-xft
 --with-imagemagick --without-gpm --with-x-toolkit=gtk3 --with-pgtk
 --with-toolkit-scroll-bars --x-includes=/usr/include
 --x-libraries=/usr/lib64 --with-libotf --with-m17n-flt --with-cairo
 --with-xwidgets --build=x86_64-suse-linux --with-dumping=pdumper
 'CFLAGS=-O2 -Wall -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3
 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables
 -fstack-clash-protection -Werror=return-type -flto=auto -g
 -D_GNU_SOURCE -DGDK_DISABLE_DEPRECATION_WARNINGS
 -DGLIB_DISABLE_DEPRECATION_WARNINGS -pipe -Wno-pointer-sign
 -Wno-unused-variable -Wno-unused-label -fno-optimize-sibling-calls
 -DPDMP_BASE='\''"emacs-wayland"'\''' LDFLAGS=-Wl,-O2'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ
IMAGEMAGICK JPEG JSON LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 MODULES
NATIVE_COMP NOTIFY INOTIFY PDUMPER PGTK PNG RSVG SECCOMP SOUND SQLITE3
THREADS TIFF TOOLKIT_SCROLL_BARS TREE_SITTER WEBP XIM XWIDGETS GTK3 ZLIB

Important settings:
  value of $LC_NUMERIC: POSIX
  value of $LC_TIME: en_US.UTF-8
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Dired by name

Minor modes in effect:
  shell-dirtrack-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  buffer-read-only: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny rfc822
mml mml-sec epa derived epg rfc6068 epg-config gnus-util
text-property-search mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums
mm-util mail-prsvr mail-utils dired-aux dired dired-loaddefs tramp-sh
tramp-cache time-stamp tramp tramp-loaddefs trampver tramp-integration
files-x tramp-compat rx shell pcomplete comint ansi-osc ring parse-time
iso8601 time-date format-spec auth-source cl-seq eieio eieio-core
cl-macs cl-loaddefs cl-lib password-cache json subr-x map byte-opt gv
bytecomp byte-compile ansi-color rmc delsel lpr easy-mmode pcase
iso-transl tooltip cconv eldoc paren electric uniquify ediff-hook
vc-hooks lisp-float-type elisp-mode mwheel term/pgtk-win pgtk-win
term/common-win pgtk-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode lisp-mode prog-mode register
page tab-bar menu-bar rfn-eshadow isearch easymenu timer select
scroll-bar mouse jit-lock font-lock syntax font-core term/tty-colors
frame minibuffer nadvice seq simple cl-generic indonesian philippine
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-v

bug#73046: 29.4; Emacs 100% CPU usage for several seconds when opening dired buffer over TRAMP

[Suhail Singh]

Eli Zaretskii  writes:

>> From: "Suhail Singh" 
>> Date: Thu, 05 Sep 2024 10:24:54 -0400
>> 
>> 
>> I observe 100% CPU usage for several seconds when invoking dired for a
>> directory on one of the remote machines I connect to.  This doesn't seem
>> to happen for another host.  The connection to the affected host is
>> slower, and on that host the issue is worse when the directory has
>> several symlinks.
>> 
>> To reproduce via =emacs -Q=:
>> - Invoke `find-file' and connect to problematic host over ssh.
>> - Once connection is established, open dired pointing to a directory on
>>   the affected host.
>> - Observe bug: it takes _several_ seconds for the dired buffer to show.
>>   During that time, Emacs CPU usage is 100%.
>> - Once dired buffer is open, invoke `revert-buffer' and observe issue
>>   again.
>> 
>> Disabling global-font-lock-mode, makes the situation better.
>> 
>> Given that rudimentary font-locking should be possible via, simply, the
>> output of `ls --dired', it is unclear why enabling font-locking makes
>> things so much worse.  If there are some aspects of font-locking that
>> perform much worse over a slower connection, it would help if there were
>> a user configuration to disable them.
>> 
>> It would also help if being over a slow connection didn't result in
>> Emacs consuming 100% of the CPU via functions such as
>> `tramp-wait-for-regexp' (based on profiler-report).  Could some of this
>> be done asynchronously?
>
> You could probably tell which parts take the time by profiling Emacs
> while it collects the Dired data, using profiler.el.  This could give
> clues about the expensive parts.  My guess would be that retrieving
> the attributes of the files Dired needs are the reason, but I could be
> wrong.

Based on =profiler-report=, the following function "chains" consume most of the
CPU:
- `font-lock-fontify-keywords-region'
  - tramp-sh-file-name-handler
- tramp-sh-handle-file-truename
  - `tramp-wait-for-regexp'
- tramp-sh-handle-file-exists-p
  - `tramp-wait-for-regexp'
- tramp-sh-handle-file-directory-p
  - `tramp-wait-for-regexp'
- tramp-sh-handle-file-attributes
  - `tramp-wait-for-regexp'

As noted previously, disabling global-font-lock-mode helps.

In related news, one thing I've observed on the affected host is that
the version of `ls' doesn't seem to yield expected output for
`ls --dired'.  Specifically, the output of `ls --dired' is the same as
the output of `ls' (i.e., `--dired' gets treated as a no-op).  The version
of `ls' on this host is:

#+begin_quote
  ls (GNU coreutils) 8.32
  Copyright (C) 2020 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later 
<https://gnu.org/licenses/gpl.html>.
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.

  Written by Richard M. Stallman and David MacKenzie.
#+end_quote

-- 
Suhail

bug#73046: 29.4; Emacs 100% CPU usage for several seconds when opening dired buffer over TRAMP

[Suhail Singh]

Eli Zaretskii  writes:

>> The version of `ls' on this host is:
>> 
>> #+begin_quote
>>   ls (GNU coreutils) 8.32
>>   Copyright (C) 2020 Free Software Foundation, Inc.
>>   License GPLv3+: GNU GPL version 3 or later 
>> .
>>   This is free software: you are free to change and redistribute it.
>>   There is NO WARRANTY, to the extent permitted by law.
>> 
>>   Written by Richard M. Stallman and David MacKenzie.
>> #+end_quote
>
> This cannot be true: the --dired option was added to 'ls' way earlier.
> I have Coreutils 6.9 from 2007 on one of my systems, and --dired is
> supported there.  To see if it is supported, try this:
>
>   $ ls -al --dired
>
> You should see 2 extra lines of output after the listing, each one
> starting with "//DIRED".

I understand that said version should have had --dired support, but
that's not what I observed.  Regardless, I got the sysadmin to do an
upgrade and now `ls --dired' is giving the expected output.  However,
the performance issues still persist and continue to be reproducible on
my end via emacs -Q.

-- 
Suhail

bug#73046: 29.4; Emacs 100% CPU usage for several seconds when opening dired buffer over TRAMP

[Suhail Singh]

Eli Zaretskii  writes:

>> Based on =profiler-report=, the following function "chains" consume most of 
>> the
>> CPU:
>> - `font-lock-fontify-keywords-region'
>>   - tramp-sh-file-name-handler
>> - tramp-sh-handle-file-truename
>>   - `tramp-wait-for-regexp'
>> - tramp-sh-handle-file-exists-p
>>   - `tramp-wait-for-regexp'
>> - tramp-sh-handle-file-directory-p
>>   - `tramp-wait-for-regexp'
>> - tramp-sh-handle-file-attributes
>>   - `tramp-wait-for-regexp'
>> 
>> As noted previously, disabling global-font-lock-mode helps.
>
> FWIW, I cannot reproduce this: I tried Dired on a remote host with
> which I have connection that is quite slow, and saw neither high CPU
> usage nor a significant delay in displaying a Dired buffer.

In my case, the directory in question had 22 symlinks.  How many
symlinks did you try with?

-- 
Suhail

bug#73046: 29.4; Emacs 100% CPU usage for several seconds when opening dired buffer over TRAMP

[Suhail Singh]

Michael Albinus  writes:

>>> Based on =profiler-report=, the following function "chains" consume most of 
>>> the
>>> CPU:
>>> - `font-lock-fontify-keywords-region'
>>>   - tramp-sh-file-name-handler
>>> - tramp-sh-handle-file-truename
>>>   - `tramp-wait-for-regexp'
>>> - tramp-sh-handle-file-exists-p
>>>   - `tramp-wait-for-regexp'
>>> - tramp-sh-handle-file-directory-p
>>>   - `tramp-wait-for-regexp'
>>> - tramp-sh-handle-file-attributes
>>>   - `tramp-wait-for-regexp'
>>>
>>> As noted previously, disabling global-font-lock-mode helps.
>>
>> FWIW, I cannot reproduce this: I tried Dired on a remote host with
>> which I have connection that is quite slow, and saw neither high CPU
>> usage nor a significant delay in displaying a Dired buffer.
>
> It seems to be related to font-locking, indeed. See variable
> `dired-font-lock-keywords'. It specifies face recognition running basic
> file oprtations. For example, ";; Broken Symbolic link" calls
> `file-truename' and `file-exists-p', while "Symbolic link to a directory"
> and ";; Symbolic link to a non-directory" invoke `file-truename' and
> `file-directory-p'.
>
> I believe it would be helpful to suppress these checks via a user
> option.

I agree.  Having a user configuration(s) that allows one to selectively
disable some of the possibly more expensive checks would be valuable.
If that were available, user configuration code could determine their
own rules or thresholds as to when those options should be toggled
(without disabling font-locking in its entirety).

If performance degradation could be dynamically checked by Emacs
(something akin to so-long-action and so-long-predicate, but for this
instance of font-locking instead) that would be ideal, but it may not be
easy to implement and it certainly isn't necessary - when things get
unresponsive, it's pretty apparent.

> And no, the checks shouldn't be suppressed for remote directories in
> general, on a fast connection they are valuable.

I agree.

> In bug#17064, the impact of these calls where discussed. The conclusion was
>
>> Or could this have any bad side effects?  Is it maybe too heavy to call
>> `file-truename'?
>
> Normally, there aren't many symlinks in a buffer, so I think the
> performance impact would be negligible.
>
> Likely, this was too optimistic ...

For the host where I noticed this issue, the directory in question had
22 symlinks.

-- 
Suhail

bug#73046: 29.4; Emacs 100% CPU usage for several seconds when opening dired buffer over TRAMP

[Suhail Singh]

Michael Albinus  writes:

>>> > FWIW, I cannot reproduce this: I tried Dired on a remote host with
>>> > which I have connection that is quite slow, and saw neither high CPU
>>> > usage nor a significant delay in displaying a Dired buffer.
>>>
>>> It seems to be related to font-locking, indeed. See variable
>>> `dired-font-lock-keywords'. It specifies face recognition running basic
>>> file oprtations. For example, ";; Broken Symbolic link" calls
>>> `file-truename' and `file-exists-p', while "Symbolic link to a directory"
>>> and ";; Symbolic link to a non-directory" invoke `file-truename' and
>>> `file-directory-p'.
>>
>> But font-lock is ON by default, so what I saw also includes this,
>> right?
>
> Sure. But we don't know the exact setting of the OP. The connection
> could be extremely slow for him, Tramp's cache might be disabled,
> whatever. Perhaps it's worth to rerun the test with "emacs -Q".

The issue was observed with emacs -Q on a remote directory that had 22
symlinks.  On the affected host, I am able to reliably reproduce the
issue by evaluating the code below and then navigating to
/tmp/test/links in dired via TRAMP.

#+name: perf-fix/tramp/font-lock-on-dired/reproducer
#+begin_src sh
  cd /tmp
  rm -rf /tmp/test
  mkdir /tmp/test
  cd /tmp/test
  for i in `seq -w 0 21`; do
  mkdir -p src/"$i"
  done
  mkdir -p links
  cd links
  for i in `seq -w 0 21`; do
  ln -sf /tmp/test/src/"$i"
  done
#+end_src

Notably, for the affected host, the issue is reproducible using above
even after a new OS install.  However, the above test case doesn't cover
all variables affecting the issue.  Specifically, when I run the above
code on a different host (with faster connection and fewer hops to get
to) and navigate to /tmp/test/links the severity of the issue is much
less.  That being said, the profiler reports (starting from emacs -Q and
after establishing connection with host in question via TRAMP) for both
the affected and not-as-affected host are similar.  I believe the issue
is present on the unaffected host as well, but with much less severity
for some reason.

My guess is that connection speed (in addition to the number of symlinks
in the directory in question) is a determining factor.  I am speculating
here, but perhaps TRAMP is busy-waiting for a response?  That would be
consistent with what I observe, but I am not sure how to verify that.

>>> I believe it would be helpful to suppress these checks via a user
>>> option.
>>
>> What's wrong with "M-x font-lock-mode RET"?
>
> You would deactivate all fontifications, not only the ones related to
> symlinks.
>
> And you must do this in a hook of the dired buffer, before fontification
> happens.

Yes, precisely.  That is my current workaround.  It's far from ideal,
but it at least allows Emacs to be in a semi-workable state.  It would
be much better if there were a way to suppress only the "expensive"
font-locking checks in this case.

> There exist `dired-hide-details-mode', which I don't know. Perhaps
> enabling it in time would do the job?

It, unfortunately, doesn't help.

-- 
Suhail

bug#73046: 29.4; Emacs 100% CPU usage for several seconds when opening dired buffer over TRAMP

[Suhail Singh]

Eli Zaretskii  writes:

> I think we should also try to establish how slow is that connection.
> How much time does "ping" take?

The server doesn't respond to "ping" requests so I cannot test that.
There are however more than 14 hops as reported by mtr.  How much more I
cannot say, unfortunately.

> How long does it take to fetch a file of, say 20 KBytes?

Given a 20kb file generated via:
#+begin_src sh :results verbatim
  mkdir -p /tmp/test
  cd /tmp/test
  dd if=/dev/urandom of=upload_test bs=10k count=2
  du -sh /tmp/test/upload_test
#+end_src

#+RESULTS:
: 20K   /tmp/test/upload_test

It takes ~4-6s to upload:
#+begin_src sh :results verbatim
  cd /tmp/test
  {
  time scp upload_test scp://${affected-host}//tmp/upload_test
  } 2>&1
#+end_src

#+RESULTS:
: 
: real  0m4.255s
: user  0m0.008s
: sys   0m0.002s

And a similar time to download:
#+begin_src sh :results verbatim
  cd /tmp/test
  {
  time scp scp://${affected-host}//tmp/upload_test ./download_test
  } 2>&1
#+end_src

#+RESULTS:
: 
: real  0m5.638s
: user  0m0.007s
: sys   0m0.000s


> Also, if font-lock is disabled, does the problem go away?

As noted in the original message:

>>> Disabling global-font-lock-mode, makes the situation better.

Things are still sluggish after disabling font-lock, but they are *much*
better (i.e., instead of a ~10s delay for revert-buffer we are down to
less than a second).  The remaining sluggishness seems reasonable given
what I know about the connection speed, but I have not done extensive
quantitative testing.  As such, while I cannot say with certainty if
"the problem" goes away entirely, or only mostly, it does improve
significantly.

-- 
Suhail

bug#73046: 29.4; Emacs 100% CPU usage for several seconds when opening dired buffer over TRAMP

[Suhail Singh]

Eli Zaretskii  writes:

> So if it takes 4 to 5 sec to move a 20KB file, then how much stuff
> needs to be moved for the Dired listing?  What does the below show, if
> you run it on that remote machine?
>
>   $ ls -al | wc

Given that the contents of the remote directory can be exactly
replicated (apart from metadata such as mtime etc) by the below code
(that I shared in another response), you should be able to answer the
question trivially:

#+name: perf-fix/tramp/font-lock-on-dired/reproducer
#+begin_src sh
  cd /tmp
  rm -rf /tmp/test
  mkdir /tmp/test
  cd /tmp/test
  for i in `seq -w 0 21`; do
  mkdir -p src/"$i"
  done
  mkdir -p links
  cd links
  for i in `seq -w 0 21`; do
  ln -sf /tmp/test/src/"$i"
  done
#+end_src

For the record, the output is as follows:

#+begin_src sh :dir /ssh:${affected-host}:/tmp
  cd /tmp/test/links
  ls -al | wc
#+end_src

#+RESULTS:
: 25 2621599


> It needs to show around 40KB to explain 10 sec of delay.

I don't understand your reasoning.  In fact if the output of ls -al was
indeed around 40kb I would have been very surprised.  The time taken for
transferring the 20KB file included initial connection costs whereas
TRAMP would presumably be reusing the same connection, but would be
sending multiple small requests.  I don't see how one can be compared to
the other, other than to say (generally) that when connection is slow
both workflows would take greater time (which is what we observe).

Note that disabling font-lock improved response delay considerably.
That means the delay is not due to transferring information contained in
`ls --dired', which is considerably fast (relatively speaking), but in
doing the additional checks that Michael mentioned:

>>> It seems to be related to font-locking, indeed. See variable
>>> `dired-font-lock-keywords'. It specifies face recognition running basic
>>> file oprtations. For example, ";; Broken Symbolic link" calls
>>> `file-truename' and `file-exists-p', while "Symbolic link to a directory"
>>> and ";; Symbolic link to a non-directory" invoke `file-truename' and
>>> `file-directory-p'.

I did some further investigation; summarizing findings below:

A. On Host-A, the network connection is fairly slow s.t. transferring a
   20KB file takes ~5s.  On Host-B, the network connection is fairly
   fast.

B. On Host-A, the time taken to refresh dired buffer containing 22
   Subdirectories (/tmp/test/src as in above code snippet) is 0.70-0.75s
   with font-lock enabled, and about the same with font-lock disabled.
   These times exclude the time taken to establish the intiial
   connection over TRAMP.

C. On Host-A, the time taken to refresh dired buffer containing 22
   symlinks (each symlink pointing to a directory, i.e., /tmp/test/links
   in the above code snippet) is 0.70-0.75s with font-lock disabled.
   With font-lock enabled the time taken is ~13-14s and the CPU is at
   100%.  These times exclude the time taken to establish the intiial
   connection over TRAMP.

D. On Host-B, the time taken to display dired buffer for /tmp/test/links
   with font-lock enabled is ~2s greater than when font-lock is
   disabled.  When /tmp/test/links contains 100 symlinks to directories
   (instead of 22), the time taken when font-lock is enabled is ~6s
   greater than when font-lock is disabled.

Given above, I conclude:

1. The issue is present when there are symlinks to directories.

2. The issue is worse when there are greater number of symlinks to
   directories.

2. The issue is worse when the connection is slower.  However, it is
   still observable when the connection is faster - if you're having
   difficulty reproducing, increase the number of symlinks to
   directories in /tmp/test/links above.

3. Given that when connection is slower, not only is the time taken for
   font-locking greater, but the CPU is at 100%, I suspect that the
   relevant code is doing some kind of busy-waiting.

The above observations seem consistent with Michael's comments above
regd. font-lock checks for "Broken Symbolink link" and "Symbolic link to
a directory".  As such, if Michael's proposal below is implemented I
believe it would be an adequate fix to the issue:

>>> I believe it would be helpful to suppress these checks via a user
>>> option. And no, the checks shouldn't be suppressed for remote
>>> directories in general, on a fast connection they are valuable.

I hope that clarifies things, and gives you sufficient information to be
able to reproduce

-- 
Suhail

bug#73046: 29.4; Emacs 100% CPU usage for several seconds when opening dired buffer over TRAMP

[Suhail Singh]

Eli Zaretskii  writes:

> Thanks you.  So the problem seems to be symlinks, and specifically
> symlinks to directories.

After sending the penultimate email, and before sending the last (which
contained the workaround for modifying `dired-font-lock-keywords'
buffer-locally) I ran some more tests, and symbolic links to files are
also problematic.

Specifically, it's the three checks marked by the following comments in
`dired-font-lock-keywords' that are problematic:

- ";; Broken Symbolic link."
- ";; Symbolic link to a directory."
- ";; Symbolic link to a non-directory."

Only when the three entries corresponding to the above are removed from
`dired-font-lock-keywords', does the issue get resolved.  Removing some
(but not all) of the entries improves matters, but doesn't resolve them
completely.  Quoting Michael:

#+begin_quote
  It seems to be related to font-locking, indeed. See variable
  `dired-font-lock-keywords'. It specifies face recognition running basic
  file oprtations. For example, ";; Broken Symbolic link" calls
  `file-truename' and `file-exists-p', while "Symbolic link to a directory"
  and ";; Symbolic link to a non-directory" invoke `file-truename' and
  `file-directory-p'.

  I believe it would be helpful to suppress these checks via a user
  option. And no, the checks shouldn't be suppressed for remote
  directories in general, on a fast connection they are valuable.
#+end_quote

> Michael, what does Tramp do specially in these cases that could
> explain the slowdown?
>
>> The above observations seem consistent with Michael's comments above
>> regd. font-lock checks for "Broken Symbolink link" and "Symbolic link to
>> a directory".
>
> Michael, what do these checks entail, and why are they so
> CPU-expensive and take a lot of time with slow connections?

-- 
Suhail

bug#73046: 29.4; Emacs 100% CPU usage for several seconds when opening dired buffer over TRAMP

[Suhail Singh]

Eli Zaretskii  writes:

>> 15 times the "test -h" command - I guess, Tramp shall do cyclic link
>> detection better.
>
> I agree.  But that only explains the time delay, no why Emacs is
> consuming 100% of CPU, right?  Waiting for the network should not
> consume CPU, unless I'm missing something.
>
> Also, Suhail Singh says that there's a significant delay when there
> are valid symlinks, in which case I don't expect Tramp to issue the
> same command 15 times, right?

Yes, while there are clearly inefficiencies in cyclic link detection,
that's not the situation for the reproducer I shared.  Font-locking
symlinks (broken, not broken; to files, to directories) trigger the
issue even without introducing cyclic references.  For what it's worth,
as I shared in an earlier exchange, the profiler-report seemed to point
the finger to `tramp-wait-for-regexp':

| Func in font-lock check | TRAMP handler   
 |
|-+--|
| `file-truename' | `tramp-sh-handle-file-truename' -> ... -> 
`tramp-wait-for-regexp'|
| `file-exists-p' | `tramp-sh-handle-file-exists-p' -> ... -> 
`tramp-wait-for-regexp'|
| `file-directory-p'  | `tramp-sh-handle-file-directory-p' -> ... -> 
`tramp-wait-for-regexp' |

Unless I misinterpreted the profiler output, something in/about
`tramp-wait-for-regexp' results in the 100% CPU usage.

-- 
Suhail

bug#73046: 29.4; Emacs 100% CPU usage for several seconds when opening dired buffer over TRAMP

[Suhail Singh]

Michael Albinus  writes:

> I believe it would be helpful to suppress these checks via a user
> option. And no, the checks shouldn't be suppressed for remote
> directories in general, on a fast connection they are valuable.
>
> I gave it a try, see appended patch.

Thank you for sharing the patch.  I haven't tested the patch yet, so my
comments below are based on reviewing the code and my experience with
the workaround I currently have.

> There's a new user option `dired-highlight-symlinks'. If non-nil (the
> default), symlinks are highlighted the same way as now. With a nil
> value, they aren't.

I propose that the non-default behaviour be that `dired-symlink-face' is
applied to the entirety of the symlink.  If you agree, it might be
better to additionally rename `dired-highlight-symlinks' accordingly to
reflect the updated behaviour.

> You can switch this option on and off globally. However, it would be
> better to do this host-wise. For this, we have connection-local
> variables. The following code snippet in your ".emacs" switches the
> option off for the remote host "remotehost".
>
> (connection-local-set-profile-variables
>  'my-dired-profile
>  '((dired-highlight-symlinks . nil)))
>
> (connection-local-set-profiles
>  '(:application tramp :machine "remotehost")
>  'my-dired-profile)
>
> Comments?

TIL about connection-local variables.  Yes, this would be helpful.
However, if so desired, I imagine that setting
`dired-highlight-symlinks' buffer-locally in the `dired-mode-hook' would
work as well.  Is my understanding correct?

-- 
Suhail

bug#73046: 29.4; Emacs 100% CPU usage for several seconds when opening dired buffer over TRAMP

[Suhail Singh]

Michael Albinus  writes:

> ;; Symbolic link to a non-directory.
> (list dired-re-sym
>   (list (lambda (end)
> - (when-let ((file (dired-file-name-at-point)))
> -   (let ((truename (ignore-errors (file-truename file
> - (and (or (not truename)
> -   (not (file-directory-p truename)))
> -   (search-forward-regexp "\\(.+-> ?\\)\\(.+\\)"
> - end t)
> + (if (not (connection-local-value dired-check-symlinks))
> + (search-forward-regexp
> +  "\\(.+-> ?\\)\\(.+\\)" end t)
> +   (when-let ((file (dired-file-name-at-point)))
> + (let ((truename (ignore-errors (file-truename file
> +   (and (or (not truename)
> + (not (file-directory-p truename)))
> + (search-forward-regexp
> + "\\(.+-> ?\\)\\(.+\\)" end t))

Minor nitpick: given the comment for this section (which seems to
suggest this only applies to symlinks to non-directories), wouldn't it
be "better" to simply have something like below instead for the if-then
case?  I.e., a separate entry in dired-font-lock-keywords for the
failsafe.

#+begin_src emacs-lisp
  ;; Generic face for symbolic links
  (list dired-re-sym
(when (not (connection-local-value dired-check-symlinks))
  '(".+" (dired-move-to-filename) nil (0 dired-symlink-face
#+end_src

This is essentially what I have in my config today, and seems to behave
as expected.

-- 
Suhail

bug#73046: 29.4; Emacs 100% CPU usage for several seconds when opening dired buffer over TRAMP

[Suhail Singh]

Michael Albinus  writes:

>>> ;; Symbolic link to a non-directory.
>>> ...
>>> + "\\(.+-> ?\\)\\(.+\\)" end t))
>>
>> Minor nitpick: given the comment for this section (which seems to
>> suggest this only applies to symlinks to non-directories), wouldn't it
>> be "better" to simply have something like below instead for the if-then
>> case?  I.e., a separate entry in dired-font-lock-keywords for the
>> failsafe.
>>
>> #+begin_src emacs-lisp
>>   ;; Generic face for symbolic links
>>   (list dired-re-sym
>> (when (not (connection-local-value dired-check-symlinks))
>>   '(".+" (dired-move-to-filename) nil (0 dired-symlink-face
>> #+end_src
>>
>> This is essentially what I have in my config today, and seems to behave
>> as expected.
>
> Possible, but I don't see that one version is "better" or "worse" than
> the other.

Understood.

> So I don't see a need to change it.

The comment ";; Symbolic link to a non-directory." should probably be
updated in that case, right?  Apologies for the noise, in case I have
misunderstood the patch.

I should be able to test out the patch later this coming week.  In case
that would be helpful, please let me know.  Thank you for adding this.

-- 
Suhail

bug#73046: 29.4; Emacs 100% CPU usage for several seconds when opening dired buffer over TRAMP

[Suhail Singh]

Eli Zaretskii  writes:

>> >> Tramp is in a loop, waiting for results from the remote side. I don't
>> >> know how to implement this differently.
>> >
>> > What kind of loop?  Can you point me to the code which loops there?
>> 
>> The umbrella function is tramp-send-command. It sends the command to
>> remote via tramp-send-string, and waits then for a proper shell prompt
>> via tramp-wait-for-output. The latter function calls
>> tramp-wait-for-regexp, which loops using tramp-accept-process-output.
>
> Did you try either calling accept-process-output with a non-zero
> timeout of, say, 5 msec, or inserting a (sit-for 0.005) into the loop
> which calls tramp-accept-process-output?
>
> Waiting for a process to respond should try not to spin without
> sleeping for a few milliseconds between polling attempts.

A constant delay busy-wait would be better than what's there today.  A
straightforward exponential back-off (start with something small like
0.005 and double on each "still-waiting" case till something larger like
0.5) could be even better.

However, I thought you'd be proposing something like registering a
"callback".  Out of curiosity, since I am unfamiliar with the relevant
Emacs internals, would it not be possible to add a function to the
equivalent of `after-change-functions' that processes the output?

-- 
Suhail

bug#73046: 29.4; Emacs 100% CPU usage for several seconds when opening dired buffer over TRAMP

[Suhail Singh]

Michael Albinus  writes:

> Tramp used a non-zero timeout in the past. This was removed some years
> ago, I don't remember the reason.

The timeout in tramp-accept-process-output was disabled in commit
54ef338ba3670415cf47fabc33a92d4904707c7e .  The commit mentions
bug#61350 , however, it's not clear (based on briefly skimming the
discussion there) that this change was ever necessary.  If not, should
the timeout be reintroduced?

> I've added (sit-for 0.005) in the loop calling
> tramp-accept-process-output. It decreases the CPU load from 100% to
> something between 45..50%, when waiting for a response from remote.

I am trying to better understand what's going on, so the following is
simply for my understanding.  Use your discretion when responding.

IIUC, we're still actively waiting for the output from the remote host,
but simple not _exclusively_ doing so thanks to the `sit-for'.  Is my
understanding correct?  If so, isn't there some mechanism to specify a
continuation that's run once the TRAMP process produces output?  Such a
mechanism shouldn't require a `sit-for' to yield control.

In other words, isn't it possible to do both font-locking and getting
the response over ssh concurrently (of the main thread, as well wrt each
other)?  If not, are there technical challenges in doing so, or simply
that it's not been implemented (and thus, possibly, we may not know what
the challenges are)?

> Pushed to master

Thank you.  I can test this (applied onto 29.4) later in the week along
with the `dired-font-lock-keywords' patch and report back.

-- 
Suhail

bug#73046: 29.4; Emacs 100% CPU usage for several seconds when opening dired buffer over TRAMP

[Suhail Singh]

Michael Albinus  writes:

> I'm happy it works stable now, and I'm reluctant to touch it w/o urgent
> need. That's why I have used the (sit-for 0.005) solution proposed by
> Eli, outside tramp-accept-process-output.

Understood; thanks for elaborating.

-- 
Suhail

bug#73046: 29.4; Emacs 100% CPU usage for several seconds when opening dired buffer over TRAMP

[Suhail Singh]

Eli Zaretskii  writes:

>> If so, isn't there some mechanism to specify a
>> continuation that's run once the TRAMP process produces output?  Such a
>> mechanism shouldn't require a `sit-for' to yield control.
>
> How would that help?  Tramp must still wait for the response to
> proceed with what it does.
>
>> In other words, isn't it possible to do both font-locking and getting
>> the response over ssh concurrently (of the main thread, as well wrt each
>> other)?
>
> Every Lisp program runs in a single thread, so how can that be done in
> parallel?
>
>> If not, are there technical challenges in doing so, or simply that
>> it's not been implemented (and thus, possibly, we may not know what
>> the challenges are)?
>
> Emacs doesn't support parallel processing, because introducing this
> into the original single-threaded design is very hard at best, due to
> a huge global state.  We have Lisp threads, but only one thread can
> run at any given time.

I was imagining a version of TRAMP that was asynchronous.  I understand
now that it isn't.

-- 
Suhail

bug#73046: 29.4; Emacs 100% CPU usage for several seconds when opening dired buffer over TRAMP

[Suhail Singh]

Michael Albinus  writes:

> FTR, some years ago (2018?) I tried to use Lisp threads with Tramp. This
> experiment failed.

OOC, the failure was from hard-to-debug edge cases while trying to get
to feature-parity or something else?

-- 
Suhail