bug#31397: 27.0.50; Emacs doesn't call package-initialize if there's no elpa directory

[Zachary Kanfer]

------------------
High-level summary:
------------------

If Emacs is started with no ~/.emacs.d/elpa directory,
#'package-initialize is not called, and I can't programatically work
with packages.

----------
Motivation
----------

I like my init file to completely set up Emacs: I want to be able to
start with an empty ~/.emacs.d/ move my init file into it, and start up
Emacs. To do that, I've had (previously to Emacs 27), this as the
preamble to my init file:

(require 'package)
(add-to-list 'package-archives
'("melpa" . "https://melpa.org/packages/"))
(package-initialize)

(unless (package-installed-p 'use-package)
(package-refresh-contents)
(package-install 'use-package))

This sets up package.el with the ability to use melpa, then installs
use-package, for use in the rest of my init file. This has worked fine.

But I recently upgraded my Emacs to a newer version built from source,
using commit 766b057e41df7316808ec7658836fda75facda75. As part of that,
I've read up on the early init file. My understanding is that users
should no longer call #'package-initialize in their init files. This is
supported by, if I do have a call to #'package-initialize, starting
Emacs gives me a warning "Warning (package): Unnecessary call to
‘package-initialize’ in init file". So I figured I would remove it. But
I quickly ran into a problem, or at least an area for better
documentation.

-------------------------
Reproduction instructions
-------------------------

A minimal init file that shows my problem is here:

(require 'package)
(add-to-list 'package-archives
'("melpa" . "https://melpa.org/packages/"))

(message "use-package is installed: %s" (package-installed-p
'use-package))

Here's how I delete the elpa directory, and then start up Emacs:

$ rm -r ~/.emacs.d/elpa
$ src/emacs

Upon doing so, here's the error I get:

Warning (initialization): An error occurred while loading
‘/home/zck/.emacs.d/init.el’:

error: package.el is not yet initialized!

To ensure normal operation, you should investigate and remove the
cause of the error in your initialization file. Start Emacs with
the ‘--debug-init’ option to view a complete error backtrace.

---------------------------
Investigation & speculation
---------------------------

I started looking around for documentation, and found this interesting
comment in simple.el's function #'command-line:

;; If any package directory exists, initialize the package system.

This seems to be the cause. I'm not sure why the package system
shouldn't be initialized if there is no package directory; I want to be
able to do package operations even if I'm starting with no package
directory. You have to start somewhere.

Thanks!

------------------
System information
------------------

In GNU Emacs 27.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.18.9)
of 2018-05-07 built on zck-laptop
Repository revision: 766b057e41df7316808ec7658836fda75facda75
Windowing system distributor 'The X.Org Foundation', version 11.0.11804000
System Description: Ubuntu 16.04.4 LTS

Recent messages:
Checking for load-path shadows...done
Auto-saving...
Overwrite mode enabled in current buffer
<pause> is undefined
Undo! [3 times]
user-error: No further undo information
C-c k is undefined
Quit
C-x C-g is undefined
Mark set

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GSETTINGS NOTIFY
ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11 THREADS LCMS2

Important settings:
value of $LANG: en_US.UTF-8
value of $XMODIFIERS: @im=ibus
locale-coding-system: utf-8-unix

Major mode: Fundamental

Minor modes in effect:
tooltip-mode: t
global-eldoc-mode: t
electric-indent-mode: t
mouse-wheel-mode: t
tool-bar-mode: t
menu-bar-mode: t
file-name-shadow-mode: t
global-font-lock-mode: t
auto-composition-mode: t
auto-encryption-mode: t
auto-compression-mode: t
buffer-read-only: t
line-number-mode: t
transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny dired dired-loaddefs
format-spec rfc822 mml mml-sec epa derived epg gnus-util rmail
rmail-loaddefs mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums
mm-util mail-prsvr mail-utils cl-extra pp cl-print package-x thingatpt
help-fns radix-tree help-mode time-date elec-pair warnings package
easymenu epg-config url-handlers url-parse auth-source cl-seq eieio
eieio-core cl-macs eieio-loaddefs password-cache json map url-vars seq
byte-opt gv bytecomp byte-compile cconv cl-loaddefs cl-lib mule-util
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode elisp-mode
lisp-mode prog-mode register page menu-bar rfn-eshadow isearch timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame cl-generic cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese composite charscript charprop case-table epa-hook jka-cmpr-hook
help simple abbrev obarray minibuffer cl-preloaded nadvice loaddefs
button faces cus-face macroexp files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote dbusbind inotify lcms2 dynamic-setting system-font-setting
font-render-setting move-toolbar gtk x-toolkit x multi-tty
make-network-process emacs)

Memory information:
((conses 16 114821 11707)
(symbols 48 21397 0)
(miscs 40 99 151)
(strings 32 34316 1830)
(string-bytes 1 888700)
(vectors 16 17166)
(vector-slots 8 524482 10750)
(floats 8 54 84)
(intervals 56 292 144)
(buffers 992 15)
(heap 1024 44657 1840))

[Zachary Kanfer]

Anything I can do to help narrow this down? I'm not sure what to do to make
this work.

[Noam Postavsky]

Post by Zachary Kanfer
I like my init file to completely set up Emacs: I want to be able to
start with an empty ~/.emacs.d/ move my init file into it, and start up
Emacs.

[...]

Post by Zachary Kanfer
I've read up on the early init file. My understanding is that users
should no longer call #'package-initialize in their init files. This is
supported by, if I do have a call to #'package-initialize, starting
Emacs gives me a warning "Warning (package): Unnecessary call to
‘package-initialize’ in init file". So I figured I would remove it. But
I quickly ran into a problem, or at least an area for better
documentation.

[...]

Post by Zachary Kanfer
(require 'package)
(add-to-list 'package-archives
'("melpa" . "https://melpa.org/packages/"))
(message "use-package is installed: %s" (package-installed-p 'use-package))
$ rm -r ~/.emacs.d/elpa
$ src/emacs
Warning (initialization): An error occurred while loading
error: package.el is not yet initialized!
Anything I can do to help narrow this down? I'm not sure what to do to make this work.

Perhaps adding Radon to Cc will help, not sure if they're subscribed to
the bug list.

[Radon Rosborough]

Thanks for the cc. I wasn't aware of this bug report.

The behavior makes sense, but is definitely undesirable. There are at
least two solutions:

1. Make `package-installed-p' and other functions automatically
initialize package.el if necessary.

2. Call `package-initialize' unconditionally during startup.

Solution (1) might make package.el functions a little slower. But
solution (2) would definitely make startup slower for people who don't
use package.el.

Solution (1) seems like the correct way to do this. Is there any
reason it wasn't done already?

Any thoughts?

[Noam Postavsky]

Post by Radon Rosborough
Thanks for the cc. I wasn't aware of this bug report.
The behavior makes sense, but is definitely undesirable. There are at
1. Make `package-installed-p' and other functions automatically
initialize package.el if necessary.
2. Call `package-initialize' unconditionally during startup.
Solution (1) might make package.el functions a little slower.

I don't think speed is an issue here. For package-installed-p in
particular, it would have no effect on speed since that function already
checks package--initialized (and checking a single variable is pretty
negligible already).

Post by Radon Rosborough
Solution (1) seems like the correct way to do this. Is there any
reason it wasn't done already?

Not sure, I might guess that a predicate like package-installed-p
shouldn't have side-effects, but package-install doesn't auto-initialize
either when called non-interactively.

Stefan, do you have any insight on this?

[Stefan Monnier]

Post by Radon Rosborough
1. Make `package-installed-p' and other functions automatically
initialize package.el if necessary.

Yes, when I changed it recently I hesitated to do that and decided to
refrain from doing so in order to minimize the changes, but it makes
a lot of sense.


Stefan


diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index 94d98178c4..56be5ef70c 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -1889,8 +1889,8 @@ package-installed-p
;; We used the quickstart: make it possible to use package-installed-p
;; even before package is fully initialized.
(memq package package-activated-list))
- ((not package--initialized) (error "package.el is not yet initialized!"))
(t
+ (unless package--initialized (package-initialize 'no-activate))
(or
(let ((pkg-descs (cdr (assq package package-alist))))
(and pkg-descs

[Zachary Kanfer]

I'm circling back around to this; I think that that change makes a lot of
sense. It seems better to me -- if you're calling one of these package
functions, you probably want to work with the package system, which is only
properly set up when you call package-initialize.

What other functions would need to call package-initialize, if it hasn't
been called? I notice that package-install already does that. Some other
functions for installing don't
(package-install-from-archive, package-install-from-buffer), but I'm not
sure whether they need to.

Post by Stefan Monnier

Post by Radon Rosborough
1. Make `package-installed-p' and other functions automatically
initialize package.el if necessary.

Yes, when I changed it recently I hesitated to do that and decided to
refrain from doing so in order to minimize the changes, but it makes
a lot of sense.
Stefan
diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index 94d98178c4..56be5ef70c 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -1889,8 +1889,8 @@ package-installed-p
;; We used the quickstart: make it possible to use package-installed-p
;; even before package is fully initialized.
(memq package package-activated-list))
- ((not package--initialized) (error "package.el is not yet
initialized!"))
(t
+ (unless package--initialized (package-initialize 'no-activate))
(or
(let ((pkg-descs (cdr (assq package package-alist))))
(and pkg-descs