• Intro
• Downloads
• Community
• Development
• News
• TALK

Improving Modest email sync reliability

Revision as of 14:14, 10 June 2010

Lots of complains have been reported regarding Modest N900 email client. Particularly when it comes to sync emails. Quoting DaveQB:

1. It takes about 20mins to check my email.
2. It uses 100% cpu for the entire time it is checking.
3. Email that is read, deleted etc are not updated on the server-side, so when I check on a different client later, nothing I did on the N900 is reflected.
4. You can't do any searching.
5. It doesn't respect subscribed folder settings on the server

I basically faced all these problems, but synchronization is the top most important one for me. This page is about trying to improve Modest email sync reliability and exploring other ways to read/write emails on N900. It mainly focuses on dealing with IMAP from a GMail account, but may be extended to other IMAP, and even other protocols such as POP.

Disclaimer

Following instructions are provided as-is. Use them as you like and at your own risk. I'm not responsible of any emails losses or anything like that. You've been warned.

Prerequisites

Before diving into details, you should be comfortable writing commands on terminal, editing files, etc... some basic Linux usage is required.

Because we'll write a lot, you may want to access your N900 through SSH. In order to do this, you'll need to deal with USB_networking between your PC and your phone.

Finally, you should know how to access your IMAP account, that is, all parameters should been known to work to avoid any doubts when testing your connection.

Accessing a IMAP GMail account using mutt

So my first try was accessing my GMail account using mutt. There have been discussions about this here, though I couldn't find a deb package file for mutt in different repository. I had to create a new one, including some needed compilation option:

• slang: allowing 256 terminal colors, reported to work better than curses
• hcache: header caching enable, preventing always accessing remote IMAP account to list emails
• imap: because we want to access an IMAP account...
• ssl: because this IMAP account is an IMAPS account (using SSL encryption)

This deb file can be found, attached to this page.

First, copy mutt deb file to your N900 (scp, mail, etc...) and install it as root:

sudo fakeroot
dpkg -i mutt_1.5.20-2_armel.deb

You then need to write your .muttrc file to setup your IMAP account connection. Here's my basic muttrc file I used to access my Gmail account:

# General
set move=no
set quit=ask-yes
set timeout=15
auto_view text/html
set realname="Sebastien Lelong"

# IMAP
set imap_authenticators="login"
set imap_passive="no"
set imap_user = 'username'
set imap_pass = 'password'
set spoolfile = imaps://imap.gmail.com:993/INBOX
set folder = imaps://imap.gmail.com:993
set record="imaps://imap.gmail.com/[Gmail]/Sent Mail"
set maildir_trash = yes
set postponed="imaps://imap.gmail.com/[Gmail]/Drafts"
set header_cache = yes
set smtp_url = smtp://username@gmail.com@smtp.gmail.com:587/
set smtp_pass = "password"

Replace username and password as needed. Launch mutt:

mutt

you'll be asked to confirm something about a certificate, then mutt will access your IMAP account and retrieve headers for all your emails. Yes, all headers (but not emails content...) ! There may be an option to only retrieve last email headers (only emails X days old), but I just don't know it. This can take quite a lot of time and kills your dataplan... Luckily this occurs only once: mutt is creating its header cache. When running again, mutt will just re-analyze its cache, locally. This can take quite some time anyway, when you have a large mailbox.

Using mutt like this is unfortunately not that nice:

• your network connection must be fast, because browsing emails will constantly make network connection to the remote IMAP account
• worst case, when you don't have network connection, you just can't read your emails, even old ones...

offlineimap to read emails when no network connection is available

While one major advantage to IMAP protocol is to being able to directly and remotely access emails on an account, when you don't have any network connection, you just can access them... A workaround is to use a tool like OfflineIMAP, which can synchronize an IMAP account with a local maildir.

This sync occurs both ways: changes on IMAP account will propagate to maildir, changes on maildir will propagate to IMAP account. This ensure what you'll locally read will also be marked as read on the IMAP server.

In order to use offlineimap, I had to get last sources including patches to select only emails not olders than X days. This will prevent getting *all* emails from the server (headers and bodies)... You'll find attached to this page a very quick & dirty deb package for offlineimap. Install it as root:

sudo fakeroot
dpkg -i offlineimap_6.2.1_all.deb
exit

Once installed, you'll need to create ~/.offlineimaprc in order to define remote IMAP and local maildir accounts. Here's mine, again minimalist. See offlineimap.conf on OfflineIMAP website to figure out possible options.

[general]
accounts = GMail
ui = Noninteractive.Basic 

[Account GMail]
localrepository = Local
remoterepository = Remote
maxage = 5
 
[Repository Local]
type = Maildir
localfolders = ~/Mail/GMail 
 
[Repository Remote]
type = Gmail
remoteuser = username
remotepass = password
realdelete = no
trashfolder = [Gmail].Trash

We basically telling offlineimap to sync a GMail IMAP account on a local maildir located in ~/Mail/GMail. Only emails younger than 5 days will be considered. All IMAP directories will be synced (no filter).

Also create maildir location:

mkdir ~/Mail/GMail

Finally, because my deb packaging isn't that good, let's create a wrapper bash script to run offlineimap:

#!/bin/bash
PYTHONPATH=/scratchbox/tools/lib/python2.3/site-packages /scratchbox/tools/bin/offlineimap 2>&1 | tee /tmp/offlineimap.log
echo Last run: `/bin/date`

Save this script in ~/bin (for instance) as "syncemail.sh", and make executable:

chmod +x syncemail.sh

Each time it's launched, you can check what it does looking as /tmp/offlineimap.log.

Time to test. Run it !

./bin/syncemail.sh

Once it's done, you can run mutt to check what it retrieved, telling to access a maildir:

mutt -f ~/Mail/GMail/INBOX

If it sounds good to you, you can program a task, for instance using Alarmed, and configure it to run this script every X minutes.

So far so good, we have our emails synced. We can read them using mutt. Still, we may like to use a graphical front-end, fully integrated to N900 system (applet, widgets, contacts, etc...).

Adding dovecot IMAP server to let Modest accessing emails locally

If only Modest could directly access our local maildir... Since it only support POP and IMAP account, we could install a local IMAP server, serving our maildir. Modest would then access this local IMAP server. This way, the remote sync part is delegated to offlineimap, while we can still use GUI front-ends. You'll still need to tell Modest to sync to the local maildir to check new emails. A short interval, 5 min, will ensure it'll be able to sync (since it's local, there's no network related issue anymore).

So, there are many IMAP server out there, I found dovecot to be easy to package and use. You'll find deb file attached to this page. Install it as root

sudo gainroot
dpkg -i dovecot_1.2.11-1_armel.deb
exit

Copy example configuration:

sudo gainroot
cp /usr/local/etc/dovecot-example.conf /usr/local/etc/dovecot.conf

Dovecot is very powerful, you can set a lot of parameters. I tried to keep it as simple as possible. Dovecot will /etc/passwd as it's user/password database. Here's my dovecot.conf file:

base_dir = /var/run/dovecot/
protocols = imap
disable_plaintext_auth = no
ssl = no
login_user = dovecot
mail_location = maildir:~/Mail/GMail/INBOX

protocol imap {
}

auth default {
 
   mechanisms = plain
 
   passdb passwd {
   }
 
   userdb passwd {
   }

   user = root
}

You can check its syntax by running:

dovecot -n

Note dovecot will serve INBOX only (more work/search to be done in order to serve multiple maildirs...). Dovecot auth will be done under user "dovecot". We need to create it:

sudo gainroot
useradd -g mail dovecot

Also, we will access maildir from Modest email client. This maildir is owned by user "user". This user doesn't have any password, but Modest won't allow you not specifying one. We thus need to set it (I'm not sure about potential side effects of setting a password for this default user. So far I haven't had any problems, but...):

sudo gainroot
passwd user

Launch dovecot, as root

sudo gainroot
/usr/local/sbin/dovecot

Test it's running with telnet:

telnet 127.0.0.1 143

should give something like:

* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE AUTH=PLAIN] Dovecot ready.

Now open Modest, and add a new account:

Account name and name: whatever
username: user
password: yourpass
IMAP server: 127.0.0.1 # "localhost" won't be recognized as legal hostname...
IMAP: 143
No secure connection

About SMTP, you can use the one you're used to use. Another improvement would be to install a small local SMTP server, and let it send its emails when it can. So far SMTP sending works with Modest, so I didn't dig further.

Conclusions

Using offlineimap and Dovecot, the synchronization process is removed from Modest email client. I'm still testing this configuration, it seems to work flawlessly. There are still issues and limitations:

• you can tell offlineimap to partially download emails, but you can't order it to fully download one specific emails (like on iPhone for instance). Attachments are also downloaded.
• some deb packaging would need to be improved. I did this quick & dirty
• you can't access different IMAP folders and different accounts. This would need to be improved, probably on dovecot side.
• "user" needs to have a password set. There can be unknown side effects. This could be avoid by setting a PAM authentication in dovecot (doesn't use /etc/passwd, no need to have a system user).
• it only deals with IMAP. Accessing POP account could be done the same way: instead of offlineimap, you could setup fetchmail or getmail and put retrieved emails in maildirs, just like here.
• ...

Hope it helps.

Cheers,

Seb