Warnings and slowness at startup due to native-compilation

trustno1 · September 8, 2022, 1:10am

What happened?

Startup time is a little bit long (1.x second) and many warnings on startup Async-native-compile-log.

What did you expect to happen?

I wanna to know whether it is normal or there is some fix I can do.

Steps to reproduce

Start Emacs.

System information


Loading data dump...

And here is the dump of startup Async-native-compile-log: autoload-warnings.log (11.1 KB)

hlissner · September 8, 2022, 11:13am

This is normal (as of 3853dff5e116). This is Emacs natively compiling your packages as they get loaded. This will only happen the first time you load each of those packages. Packages will need to be recompiled if you up/downgrade Emacs, however (or if you delete the native bytecode in ~/.emacs.d/.local/cache/eln/).

But since native-comp appears to have stability issues on and off, from version to version of Emacs, I am rethinking the change, and/or looking for a way for users to reliably toggle it.

rahguzar · September 8, 2022, 2:02pm

Having a way to toggle will be a nice. After the updates yesterday, I had emacs lockup on startup quite a few times and I wasn’t able to C-g out of it. Sending USR2 sometimes worked but only after sending it a few times but the backtrace produced contained nothing relevant, most of the time it consisted of just a substitute-command-keys("Quit"). A couple times the lokcup was accompanied by runaway memory usage but most of the times not.

I don’t have any evidence that this was related to native compilation except that emacs eventually stabilized itself and now I can open it without locking it up and I assume that this is because everything I use has been compiled.

hlissner · September 8, 2022, 3:33pm

Try adding this to $DOOMDIR/init.el:

(setq native-comp-async-jobs-number 1)

It will limit the number of jobs native-comp will do at a time; thus limiting the memory consumption and load on the system. It’s something I’m considering mainlining, given the similar complaints I’ve heard about it so far.

trustno1 · September 9, 2022, 2:14am

Thanks for your reply, it helps a lot!

p.s. The .emacs.d/.local/autoload.el is compiled everytime I launch Emacs, is that normal too? Even no any package being updated ever since?

rahguzar · September 9, 2022, 9:41am

hlissner:

Try adding this to $DOOMDIR/init.el :
(setq native-comp-async-jobs-number 1)
It will limit the number of jobs native-comp will do at a time; thus limiting the memory consumption and load on the system. It’s something I’m considering mainlining, given the similar complaints I’ve heard about it so far.

The situation happened to me again because ran doom build -r thinking it will just build the changed packages (for me it used to build just the local packages) but it built everything. So when I restarted emacs the problem of locking up started happening again. I had to use USR2 a lot of times but I did finally manage to get a decent backtrace which lead me to believe that the problem is with doom’s incremental loading of package. The way lock up happens, just a little after the startup also makes me think that is the reason since incremental loading happens on idle timers.

Putting (setq doom-incremental-packages nil) in my config fixed the issue. I updated doom which rebuilt all my packages after this change and now emacs starts fine. I can see native compilation happening in the background and it is seamless.

The lockup were really bad. In one case the emacs consumed about 10GB of my ram really quickly before I sent SIGTERM to it, but mostly it was just emacs continuously hogging one of my computer’s cores.

hlissner · September 9, 2022, 2:38pm

While incremental loading is certainly a contributor, it is not the root of the problem. Are you on a PGTK build of Emacs, by chance? I find this to be the common denominator with folks complaining about native-comp lock ups and, if so, I’d highly recommend moving back to 28.1 stable, or at least a non-pgtk build. I suspect it will address the runaway memory usage as well.

rahguzar · September 9, 2022, 2:56pm

I am indeed on a PGTK build specifically this one Show home:agraul / emacs-pgtk-nativecomp - openSUSE Build Service which is a backport to emacs 28.

PGTK is the only one that works on wayland and I am not feeling any pain after disabling incremental loading so I think I will stick to it for now. But I will keep this in mind if/when I encounter future problems, so thanks a lot.

seagull · September 9, 2022, 4:09pm

Hi, PGTK is not an option of the emacs-28 branch, only the master branch. 28.1 and 28.2 both (with nativecomp) showed the unwanted behavior in my case. I will try building an 28.2 without nativecomp option.

hlissner · September 10, 2022, 10:30am

This issue should be resolved as of this commit:

github.com/doomemacs/doomemacs

fix: memory leak & freezes on native-comp+pgtk builds

committed 11:38PM - 09 Sep 22 UTC

hlissner

+39 -42

b7f84bd introduced a nasty regression that caused an infinite loop and runaway m…emory usage on some pgtk+native-comp builds of Emacs when it attempted to perform deferred native compilation of your packages. This would make Emacs unusable and, if left alone, could even crash your system. The only Emacs builds I'm certain are affected are derived from flatwhatson/emacs (like emacs-pgtk-native-comp on Guix and Arch Linux in particular). 28.1 stable and master (on emacs-mirror/emacs@e13509468b7c) are unaffected. It appears that some, earlier pgtk builds stack idle timers differently. I'm not entirely sure how, because it doesn't manifest in more recent builds of Emacs, and I'm already burnt out on debugging this, but here's how Doom encountered it: Doom has an incremental package loader; it loads packages, piecemeal, after Emacs has been idle for 2s, then again every 0.75s until it finishes or the user sends input (then it waits another 2s before starting again). However, if at any time the iloader detected that native-compilation is in progress, it waits 2s before trying again (repeat, until native-comp is done). But here's the catch, given the following: (run-with-idle-timer 2 nil (lambda () (run-with-idle-timer 1 nil (lambda () (message "hi"))))) I had assumed "hi" would be emitted after 3 seconds (once idle), but instead it is emitted after 2. Like this, Doom's iloader would elapse one idle timer directly into another, ad infinitum, until Emacs was forcibly killed. By switching to run-at-time and employing my own rudimentary idle timer, I avoid this issue. Also, the iloader no longer needs to be considerate of native-comp, because the latter does its own rate-limiting controlled by native-comp-async-jobs-number. Amend: b7f84bdd0105

Update Doom and you should no longer see these lockups, freezes, or runaway memory usage, even without the fixes I’ve suggested earlier in this thread (I’d recommend doing away with all of them, in fact).

rahguzar · September 11, 2022, 10:09pm

Thanks a lot! I can confirm there are no more lockups after updating to the current commit.

system · September 25, 2022, 10:10pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.