Discussion:
bug#19607: issue with Emacs 24.4.1 but not with 24.3.1 : changed on disk; really edit the buffer
(too old to reply)
Bertrand Brelier
2015-01-15 13:23:37 UTC
Permalink
Hello everybody,

I am using emacs on a Ubuntu server (14.04) to edit files.

I am editing files that are located in an encrypted directory with encfs
located on a Microsoft server (accessed with cifs).

When I edit the file and save it, and I continue to edit the file (without
closing it), I have the following issue where emacs thinks that the file
has changed on disk :

changed on disk; really edit the buffer? (y, n, r or C-h)
has changed since visited or saved. Save anyway? (yes or no)

This problem is reproducible (it actually happens every-time) but only
happens with Emacs 24.4.1 and not with Emacs 24.3.1.

Also, I do not have the problem when I use an encrypted directory locally
(not on an external server) or when I use an non-encrypted directory on the
Microsoft server.

I have also checked that the file is not used or modified by another user
or program and since I do not have any issue with the version 24.3.1 on the
same files, I am wondering if the issue could come from the version 24.4.1

Please let me know if you need further information to investigate the issue.

Thank you for your help, I really appreciate working with emacs.

Regards,

Bertrand
Paul Eggert
2015-01-20 09:39:26 UTC
Permalink
If memory serves, Emacs 24.4 is a bit more careful to detect race conditions
when a file changes while you're editing it, and most likely this is running
afoul of some aspect of encfs. I suggest running 'strace -o emacs.tr emacs -Q',
reproducing the problem, and looking at the resulting trace file. You can do
the same thing with both old and new Emacs and see how the traces differ. You
can email the traces to ***@debbugs.gnu.org. We don't need the full traces,
just the parts relevant to accessing the files in question, notably the file
status and time stamps.
Bertrand Brelier
2015-01-20 13:19:29 UTC
Permalink
Dear Paul,

Thank you for your help.

I reproduced the problem and saved the trace files in the attached tar
file.
I edited two files with the 2 emacs versions with the commands :

strace -o emacs24.4.1.tr emacs-24.4 -Q Test24.4.1.txt
strace -o emacs24.3.1.tr emacs24 -Q Test24.3.1.txt

I was not sure how to just select the file access, so I provided the full
trace files.

Let me know if you need anything-else.

Regards,

Bertrand
Post by Paul Eggert
If memory serves, Emacs 24.4 is a bit more careful to detect race
conditions when a file changes while you're editing it, and most likely
this is running afoul of some aspect of encfs. I suggest running 'strace
-o emacs.tr emacs -Q', reproducing the problem, and looking at the
resulting trace file. You can do the same thing with both old and new
Emacs and see how the traces differ. You can email the traces to
relevant to accessing the files in question, notably the file status and
time stamps.
Paul Eggert
2015-01-21 00:21:54 UTC
Permalink
Post by Bertrand Brelier
Let me know if you need anything-else.
Thanks, those traces lack time stamps which are at the heart of the
problem. Can you please rerun both tests with time stamp details in the
stat-related system calls? Something like the following shell command
should do it:

strace -o /tmp/tr -e abbrev=\!stat,fstat,lstat emacs -Q filename

Depending on your shell, you may need to omit the backslash.
Bertrand Brelier
2015-01-21 13:12:25 UTC
Permalink
Hello Paul,

Thanks for your help,

Here are the files with the time stamps included in the traces.

Cheers,

Bertrand
Post by Paul Eggert
Post by Bertrand Brelier
Let me know if you need anything-else.
Thanks, those traces lack time stamps which are at the heart of the
problem. Can you please rerun both tests with time stamp details in the
stat-related system calls? Something like the following shell command
strace -o /tmp/tr -e abbrev=\!stat,fstat,lstat emacs -Q filename
Depending on your shell, you may need to omit the backslash.
Paul Eggert
2015-01-22 02:20:27 UTC
Permalink
Sorry, I'm still having trouble seeing what the bug is. strace doesn't output
full time stamps; it tells us their values only to the nearest second.
Can you run Emacs under a debugger? Here's a recipe, if you have the time:

$ wget ftp://ftp.gnu.org/gnu/emacs/emacs-24.4.tar.xz
$ tar xf emacs-24.4.tar.xz
$ cd emacs-24.4
$ ./configure CFLAGS='-g3 -O0'
$ make
$ cd src
$ gdb emacs
(gdb) source .gdbinit
(gdb) b fileio.c:5338
(gdb) r
[ Now edit the file in Emacs, until the breakpoint is hit. ]
(gdb) p mtime
(gdb) p b->modtime

I just now tried all that and got:

(gdb) p mtime
$5 = {
tv_sec = 1421893112,
tv_nsec = 705816459
}
(gdb) p b->modtime
$6 = {
tv_sec = 1421893112,
tv_nsec = 705816459
}

which is what I'd expect: the last-modified time of the file (mtime) equals the
buffer's opinion of it (b->modtime). What do you get?
Bertrand Brelier
2015-01-22 14:16:26 UTC
Permalink
Hello Paul,

I followed the steps but as soon as I try to edit the file when using gdb,
the emacs windows freezes and I cannot do anything (I tried to open a local
file, not on the network, with no encryption).

Here is what I did :
pwd
/home/bertrand/Emacs/emacs-24.4/src

gdb ./emacs
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./emacs...done.
warning: File "/home/bertrand/Emacs/emacs-24.4/src/.gdbinit" auto-loading
has been declined by your `auto-load safe-path' set to
"$debugdir:$datadir/auto-load".
To enable execution of this file add
add-auto-load-safe-path /home/bertrand/Emacs/emacs-24.4/src/.gdbinit
line to your configuration file "/home/bertrand/.gdbinit".
To completely disable this security protection add
set auto-load safe-path /
line to your configuration file "/home/bertrand/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual. E.g., run from the
shell:
info "(gdb)Auto-loading safe path"
(gdb) source .gdbinit
SIGINT is used by the debugger.
Are you sure you want to change it? (y or n) [answered Y; input not from
terminal]
DISPLAY = localhost:10.0
TERM = xterm
Breakpoint 1 at 0x5432ba: file emacs.c, line 351.
Temporary breakpoint 2 at 0x56507f: file sysdep.c, line 850.
(gdb) b fileio.c:5338
Breakpoint 3 at 0x58e027: file fileio.c, line 5338.
(gdb) r
Starting program: /home/bertrand/Emacs/emacs-24.4/src/emacs
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffea11c700 (LWP 2263)]

** (emacs:2259): WARNING **: Couldn't connect to accessibility bus: Failed
to connect to socket /tmp/dbus-eho0AgRKqo: Connection refused
[New Thread 0x7fffe928d700 (LWP 2265)]
[New Thread 0x7fffe8a8c700 (LWP 2267)]
[New Thread 0x7fffd80a7700 (LWP 2268)]
[New Thread 0x7fffd78a6700 (LWP 2269)]
[Thread 0x7fffd78a6700 (LWP 2269) exited]
[New Thread 0x7fffd78a6700 (LWP 2270)]
[Thread 0x7fffd80a7700 (LWP 2268) exited]
[New Thread 0x7fffd80a7700 (LWP 2272)]
[Thread 0x7fffd78a6700 (LWP 2270) exited]

Breakpoint 3, Fverify_visited_file_modtime (buf=29001941) at fileio.c:5338
5338 if (timespec_cmp (mtime, b->modtime) == 0
(gdb)
(gdb) quit
A debugging session is active.

Inferior 1 [process 2259] will be killed.

Quit anyway? (y or n) y


I had to quit the gdb session as the emacs windows froze (repeated the
procedure 3 times and it froze each time).

I thought the problem could come from the bus warning but without gdb I
have the same warning but no issue to edit the same file :
./emacs

** (emacs:2275): WARNING **: Couldn't connect to accessibility bus: Failed
to connect to socket /tmp/dbus-eho0AgRKqo: Connection refused

Any suggestions ?

Thank you for your help,

Cheers,

Bertrand
Post by Paul Eggert
Sorry, I'm still having trouble seeing what the bug is. strace doesn't
output full time stamps; it tells us their values only to the nearest
second.
$ wget ftp://ftp.gnu.org/gnu/emacs/emacs-24.4.tar.xz
$ tar xf emacs-24.4.tar.xz
$ cd emacs-24.4
$ ./configure CFLAGS='-g3 -O0'
$ make
$ cd src
$ gdb emacs
(gdb) source .gdbinit
(gdb) b fileio.c:5338
(gdb) r
[ Now edit the file in Emacs, until the breakpoint is hit. ]
(gdb) p mtime
(gdb) p b->modtime
(gdb) p mtime
$5 = {
tv_sec = 1421893112,
tv_nsec = 705816459
}
(gdb) p b->modtime
$6 = {
tv_sec = 1421893112,
tv_nsec = 705816459
}
which is what I'd expect: the last-modified time of the file (mtime)
equals the buffer's opinion of it (b->modtime). What do you get?
Paul Eggert
2015-01-22 20:55:11 UTC
Permalink
Post by Bertrand Brelier
Hello Paul,
I followed the steps but as soon as I try to edit the file when using
gdb, the emacs windows freezes and I cannot do anything (I tried to
open a local file, not on the network, with no encryption).
That's interesting, as it worked for me. We should try to see what differs.

Perhaps it's something involved with your startup. Can you try running
Emacs with the -Q option instead? That is, instead of this:

(gdb) r

do this:

(gdb) r -Q

Sorry if this all seems slow and painstaking, but that's life with
remote debugging.

For what it's worth, I tried to reproduce the problem with encfs on my
machine (Fedora 21 x86-64) with a local directory, and could not
reproduce it.
Bertrand Brelier
2015-01-22 21:26:07 UTC
Permalink
Hello Paul,

Thanks for your help and patience.
I tried with r- Q but same issue, emacs freezes when I start editing the
file (I can open it but not modify its content), For this test, the file is
stored locally (no remote cifs access, no encryption),

I do not have the problem with encfs on a local directory, only when the
encrypted directory is stored on another server accessed with cifs. I do
not have any problem with a remote directory accessed with cifs it if is
not encrypted (I only have the issue when I use an encrypted directory with
cifs).

Cheers,

Bertrand
Post by Paul Eggert
Post by Bertrand Brelier
Hello Paul,
I followed the steps but as soon as I try to edit the file when using
gdb, the emacs windows freezes and I cannot do anything (I tried to open a
local file, not on the network, with no encryption).
That's interesting, as it worked for me. We should try to see what differs.
Perhaps it's something involved with your startup. Can you try running
(gdb) r
(gdb) r -Q
Sorry if this all seems slow and painstaking, but that's life with remote
debugging.
For what it's worth, I tried to reproduce the problem with encfs on my
machine (Fedora 21 x86-64) with a local directory, and could not reproduce
it.
Paul Eggert
2015-01-22 22:42:37 UTC
Permalink
Post by Bertrand Brelier
I tried with r- Q but same issue, emacs freezes when I start editing
the file (I can open it but not modify its content)
Ah, sorry, I wasn't explicit enough. How about this. When Emacs
freezes, print the requested values in the debugger and then continue
with the "c" command. I just now did that, with the following results
for me:

$ gdb ./emacs
(gdb) source .gdbinit
(gdb) b fileio.c:5338
(gdb) r -Q
[In emacs, type C-x C-f abcdef RET, then type "x".]
Breakpoint 3, Fverify_visited_file_modtime (buf=27596677) at fileio.c:5338
5338 if (timespec_cmp (mtime, b->modtime) == 0
(gdb) p mtime
$1 = {
tv_sec = 1421966150,
tv_nsec = 791881570
}
(gdb) p b->modtime
$2 = {
tv_sec = 1421966150,
tv_nsec = 791881570
}
(gdb) c
Continuing.
[In emacs, type C-x C-s.]

Breakpoint 3, Fverify_visited_file_modtime (buf=27596677) at fileio.c:5338
5338 if (timespec_cmp (mtime, b->modtime) == 0
(gdb) p mtime
$3 = {
tv_sec = 1421966150,
tv_nsec = 791881570
}
(gdb) p b->modtime
$4 = {
tv_sec = 1421966150,
tv_nsec = 791881570
}
(gdb) c
Continuing.

Breakpoint 3, Fverify_visited_file_modtime (buf=27596677) at fileio.c:5338
5338 if (timespec_cmp (mtime, b->modtime) == 0
(gdb) p mtime
$5 = {
tv_sec = 0,
tv_nsec = -1
}
(gdb) p b->modtime
$6 = {
tv_sec = 1421966150,
tv_nsec = 791881570
}
(gdb) c
Continuing.
[At this point, Emacs says "Wrote /home/eggert/decrypted/abcdef.]
Bertrand Brelier
2015-01-23 13:15:48 UTC
Permalink
Hello Paul,

Sorry, I have not used gdb for a while a forgot about asking the debugger
to continue.

I followed your request and you can see the log file attached to this email.

I modified the file 2 times : the first time, no issue I can save the file
but when I try to modify it a second time, emacs tells me :changed on disk;
really edit the buffer .....

Thanks for your help,

Cheers,

Bertrand
Post by Bertrand Brelier
I tried with r- Q but same issue, emacs freezes when I start editing the
file (I can open it but not modify its content)
Ah, sorry, I wasn't explicit enough. How about this. When Emacs freezes,
print the requested values in the debugger and then continue with the "c"
$ gdb ./emacs
(gdb) source .gdbinit
(gdb) b fileio.c:5338
(gdb) r -Q
[In emacs, type C-x C-f abcdef RET, then type "x".]
Breakpoint 3, Fverify_visited_file_modtime (buf=27596677) at fileio.c:5338
5338 if (timespec_cmp (mtime, b->modtime) == 0
(gdb) p mtime
$1 = {
tv_sec = 1421966150,
tv_nsec = 791881570
}
(gdb) p b->modtime
$2 = {
tv_sec = 1421966150,
tv_nsec = 791881570
}
(gdb) c
Continuing.
[In emacs, type C-x C-s.]
Breakpoint 3, Fverify_visited_file_modtime (buf=27596677) at fileio.c:5338
5338 if (timespec_cmp (mtime, b->modtime) == 0
(gdb) p mtime
$3 = {
tv_sec = 1421966150,
tv_nsec = 791881570
}
(gdb) p b->modtime
$4 = {
tv_sec = 1421966150,
tv_nsec = 791881570
}
(gdb) c
Continuing.
Breakpoint 3, Fverify_visited_file_modtime (buf=27596677) at fileio.c:5338
5338 if (timespec_cmp (mtime, b->modtime) == 0
(gdb) p mtime
$5 = {
tv_sec = 0,
tv_nsec = -1
}
(gdb) p b->modtime
$6 = {
tv_sec = 1421966150,
tv_nsec = 791881570
}
(gdb) c
Continuing.
[At this point, Emacs says "Wrote /home/eggert/decrypted/abcdef.]
Paul Eggert
2015-01-23 22:27:25 UTC
Permalink
Thanks, the key problem is the fourth time the breakpoint is hit, where
the output in your log looks like this:

Breakpoint 3, Fverify_visited_file_modtime (buf=27021317) at fileio.c:5338
5338 if (timespec_cmp (mtime, b->modtime) == 0
(gdb) p mtime
$7 = {
tv_sec = 1422018451,
tv_nsec = 132061000
}
(gdb) p b->modtime
$8 = {
tv_sec = 1422018451,
tv_nsec = 40059000
}

The two time stamps should be the same, but the nanoseconds component
differ (the tv_sec components are the same, which is why we don't
observe any bugs in the strace output).

I suspect a bug in the file system. Can you please try running the
attached program, with the current directory being that file system? It
should take about 2 seconds. You might try running it twice (it may
depend on whether the file already exists).
Bertrand Brelier
2015-01-26 13:08:23 UTC
Permalink
Hello Paul,

I ran your code (twice to make sure) and in both cases it returned :

last-modified times do not conform to:
http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html#tag_04_08

Cheers,

Bertrand
Post by Paul Eggert
Thanks, the key problem is the fourth time the breakpoint is hit, where
Breakpoint 3, Fverify_visited_file_modtime (buf=27021317) at fileio.c:5338
5338 if (timespec_cmp (mtime, b->modtime) == 0
(gdb) p mtime
$7 = {
tv_sec = 1422018451,
tv_nsec = 132061000
}
(gdb) p b->modtime
$8 = {
tv_sec = 1422018451,
tv_nsec = 40059000
}
The two time stamps should be the same, but the nanoseconds component
differ (the tv_sec components are the same, which is why we don't observe
any bugs in the strace output).
I suspect a bug in the file system. Can you please try running the
attached program, with the current directory being that file system? It
should take about 2 seconds. You might try running it twice (it may depend
on whether the file already exists).
Paul Eggert
2015-01-26 19:43:49 UTC
Permalink
OK, thanks, then this is definitely not a GNU Emacs bug, since the
problem can be reproduced by the small C test program without involving
Emacs. I have filed a bug report for encfs here:

https://github.com/vgough/encfs/issues/52
Paul Eggert
2015-01-27 22:29:57 UTC
Permalink
I see from Jakob's email that the bug can be reproduced on CIFS without
EncFS. However, there's still something wrong here, as Emacs has code
to work around the CIFS bug; see the comment about CIFS in
emacs/src/fileio.c starting at line 4897, here:

http://git.savannah.gnu.org/cgit/emacs.git/tree/src/fileio.c?id=a56eab8259568ea1389e972623e46359e73c0233#n4897

Unfortunately, this workaround for the CIFS bug does not appear to
suffice for EncFS over CIFS.

Bertrand, can you please try the attached program, both on plain CIFS
and on EncFS over CIFS? It attempts to mimic Emacs's actions more
precisely than the previous test program, which may help us to narrow
down the problem. Thanks.
Jakob Unterwurzacher
2015-01-27 23:04:26 UTC
Permalink
Looks like that's it. I have added printfs so I understand what is
going on:

CIFS:

Emacs should work around this POSIX-conformance bug.
stat #1: 1422398418.561215369
stat #2: 1422398418.561215300
stat #3: 1422398418.561215300


CIFS + encfs:

Emacs does not work around this POSIX-conformance bug.
stat #1: 1422398433.735274107
stat #2: 1422398433.735274107
stat #3: 1422398433.735274100


CIFS + "encfs -o attr_timeout=0":

Emacs should work around this POSIX-conformance bug.
stat #1: 1422398527.607547169
stat #2: 1422398527.607547100
stat #3: 1422398527.607547100


The kernel caches FUSE stat() values for 1 second by default, this
defeats emacs' workaround.
Passing "-o attr_timeout=0" to encfs disables that cache.
Paul Eggert
2015-01-28 04:56:26 UTC
Permalink
The kernel caches FUSE stat() values for 1 second by default, this defeats
emacs' workaround.
Passing "-o attr_timeout=0" to encfs disables that cache.
Looking at the fuse man page, it appears that if one uses EncFS atop a network
file system, the mount should not use the kernel_cache option, and should use
the auto_cache option or perhaps ac_attr_timeout=0 (the man page is unclear
about all this, unfortunately). Perhaps EncFS should use appropriate settings
for these options by default if it determines that the underlying file system is
network-based.

Anyway, assuming "-o attr_timeout=0" (or something similar) solves the problem,
we do have a workaround now.
Glenn Morris
2015-03-23 18:58:11 UTC
Permalink
So does Emacs need to do anything about this, or should this be closed?
Paul Eggert
2015-03-24 00:13:07 UTC
Permalink
Post by Glenn Morris
So does Emacs need to do anything about this, or should this be closed?
I'll close it. It's an open bug in encfs:

https://github.com/vgough/encfs/issues/52

and the bug needs to be fixed there regardless of what Emacs does. I don't see
an easy way for Emacs to work around the bug.

Bertrand Brelier
2015-01-28 13:11:45 UTC
Permalink
Hello Paul,

I tested the new code :

plain CIFS :
Emacs should work around this POSIX-conformance bug.

on EncFS over CIFS :
Emacs does not work around this POSIX-conformance bug.

Thanks for your help,

Cheers,

Bertrand
Post by Paul Eggert
I see from Jakob's email that the bug can be reproduced on CIFS without
EncFS. However, there's still something wrong here, as Emacs has code to
work around the CIFS bug; see the comment about CIFS in emacs/src/fileio.c
http://git.savannah.gnu.org/cgit/emacs.git/tree/src/fileio.c?id=
a56eab8259568ea1389e972623e46359e73c0233#n4897
Unfortunately, this workaround for the CIFS bug does not appear to suffice
for EncFS over CIFS.
Bertrand, can you please try the attached program, both on plain CIFS and
on EncFS over CIFS? It attempts to mimic Emacs's actions more precisely
than the previous test program, which may help us to narrow down the
problem. Thanks.
Jakob Unterwurzacher
2015-01-26 23:08:15 UTC
Permalink
Encfs contributor here.

Interesting problem.
Does the test program run on the plain cifs mount (without encfs in
between)
without complaining?
Jakob Unterwurzacher
2015-01-27 21:30:58 UTC
Permalink
This can also be reproduced on CIFS (without EncFS).
Forwarded to https://bugzilla.samba.org/show_bug.cgi?id=11078
Loading...