`mu4e` spontaneously stopped syncing emails

[fosskers]

What happened?

About two weeks ago or so, mu4e stopped syncing emails properly. When I b u within a *mu4e* buffer, I can see in the little bottom popup that it downloads the remote emails, but then the UI fails to increase my Unread Messages count, and the number of in store messages is stuck at its old number.

Interestingly, a further b u doesn’t download more messages. So something down in the pipes is aware of what messages it has or doesn’t have, and yet this information isn’t being propagated to the UI anymore.

Upon sending a message, I’m warned:

error in process filter: mu4e-error-handler: Error 106: failed to add message: add_or_update_msg: xapian error ‘Db block overwritten - are there multiple writers?’

And the UI enters a weird state. My sent email does actually go through to the recipient, but there’s no trace of it in mu’s own SENT folder.

What did you expect to happen?

I expect to see my new email messages and send emails without issue.

Steps to reproduce

1. SPC o m
2. b u
3. Notice that new messages aren’t shown.

System information

Note that I’m running Emacs HEAD (or close to it) and the most recent version of mu4e, 1.6.10.

Loading data dump...

[Lejon]

Just to check (and to try out discourse ):

• is mbsync (or whatever you’re using) downloading the messages
• does running mu index work as expected?
• does other mu based email clients work (like mug)?
• does it work in the doom sandbox with Doom + modules - your private config?

[fosskers]

@Lejon

is mbsync (or whatever you’re using) downloading the messages

It seems to be.

❯ mbsync --all
C: 1/1  B: 1/1  F: +0/0 *0/0 #0/0  N: +3/3 *4/4 #0/0

@Lejon

does running mu index work as expected?

❯ mu index
indexing maildir /home/colin/.mail -> store /home/colin/.cache/mu/xapian

** (mu:22674): CRITICAL **: 13:18:21.757: remove_messages: xapian error 'Expected block 1580 to be level 1, not 2'
| indexing messages; processed: 24; updated/new: 0; cleaned-up: 33

@Lejon

does other mu based email clients work (like mug )?

I haven’t tried any other client.

@Lejon

does it work in the doom sandbox with Doom + modules - your private config ?

Given that it requires a decent amount of custom config to work, would such a test even be possible?

[jeetelongname]

This is what I get when I run mu index

chattertop:(Mon 18) ᐅ mu index                                                                                      
indexing maildir /home/jeet/.mail -> store /home/jeet/.cache/mu/xapian
| indexing messages; processed: 0; updated/new: 0; cleaned-up: 0

maybe try reinitialising the database?

> mu init --maildir ~/.mail --my-address email@example.com # replace as needed

> mu index

[fosskers]

Reinitializing as suggested above fixed it, and I didn’t lose any email data! I guess there are two underlying databases?

[jeetelongname]

honestly it could be a number of things, corruptions when downloaded, conflicts when indexing, butterfly’s in Brazil flipping bits on your drive.

when you say 2 underlying databases do you mean does mu use 2 or does a conflict occur because 2 db’s were made?

[fosskers]

I mean that it seems like mbsync is doing one thing, and then mu feeds off that and creates its own index. My problem was on the mu end; somehow its index got corrupted (or something).

[jeetelongname]

I am not terribly sure what caused it and I don’t think I would be qualified to guess. all I know is that these things do happen from time to time because networks and stuff

[liam]

Yes. mbsync syncs a remote IMAP store with a local maildir, which is just a directory containing email files organized in a standard way.

mu takes a local maildir and generates a binary index (via the Xapian library) to expedite full-text search. It doesn’t contain any critical data. notmuch is similar, except the tags complicate things, because they’re a notmuch-specific extension. It’s analogous to org-roam's database, which is simply an index expediting things like backlink discovery. They’re just for performance.

It looks like you encountered a mu bug here. I’m not sure what caused it, but fortunately, for the reasons above, it’s easy enough to just rebuild the database.

I’m a big fan of the “all data in plain text, indexed or exported to read-only formats when necessary” approach, because plain text is pretty resilient, especially when tracked in version control. Another tool I’m interested in that uses this model is public-inbox.