Editing Improving Modest email sync reliability

Warning: You are not logged in. Your IP address will be recorded in this page's edit history.
The edit can be undone. Please check the comparison below to verify that this is what you want to do, and then save the changes below to finish undoing the edit.
Latest revision Your text
Line 1: Line 1:
-
Lots of complains have been reported regarding Modest [[Nokia N900|N900]] email client. Particularly when it comes to sync emails. Quoting [http://talk.maemo.org/showthread.php?t=37225 DaveQB]:
+
Lots of complains have been reported regarding Modest N900 email client. Particularly when it comes to sync emails. Quoting [http://talk.maemo.org/showthread.php?t=37225 link title DaveQB]:
# It takes about 20mins to check my email.
# It takes about 20mins to check my email.
Line 6: Line 6:
# You can't do any searching.
# You can't do any searching.
# It doesn't respect subscribed folder settings on the server
# It doesn't respect subscribed folder settings on the server
-
 
I basically faced all these problems, but synchronization is the top most important one for me. This page is about trying to improve Modest email sync reliability and exploring other ways to read/write emails on N900. It mainly focuses on dealing with IMAP from a GMail account, but may be extended to other IMAP, and even other protocols such as POP.
I basically faced all these problems, but synchronization is the top most important one for me. This page is about trying to improve Modest email sync reliability and exploring other ways to read/write emails on N900. It mainly focuses on dealing with IMAP from a GMail account, but may be extended to other IMAP, and even other protocols such as POP.
Line 31: Line 30:
* ssl: because this IMAP account is an IMAPS account (using SSL encryption)
* ssl: because this IMAP account is an IMAPS account (using SSL encryption)
-
This deb file can be found, see links section at the end of this page.
+
This deb file can be found, attached to this page.
-
First, copy mutt deb file to your N900 (scp, mail, etc...) and install it [[root access|as root]]:
+
First, copy mutt deb file to your N900 (scp, mail, etc...) and install it as root:
-
  sudo gainroot
+
  sudo fakeroot
  dpkg -i mutt_1.5.20-2_armel.deb
  dpkg -i mutt_1.5.20-2_armel.deb
You then need to write your .muttrc file to setup your IMAP account connection. Here's my basic muttrc file I used to access my Gmail account:
You then need to write your .muttrc file to setup your IMAP account connection. Here's my basic muttrc file I used to access my Gmail account:
-
<pre>
 
-
# General
 
-
set move=no
 
-
set quit=ask-yes
 
-
set timeout=15
 
-
auto_view text/html
 
-
set realname="Sebastien Lelong"
 
-
# IMAP
+
# General
-
set imap_authenticators="login"
+
set move=no
-
set imap_passive="no"
+
set quit=ask-yes
-
set imap_user = 'username'
+
set timeout=15
-
set imap_pass = 'password'
+
auto_view text/html
-
set spoolfile = imaps://imap.gmail.com:993/INBOX
+
set realname="Sebastien Lelong"
-
set folder = imaps://imap.gmail.com:993
+
-
set record="imaps://imap.gmail.com/[Gmail]/Sent Mail"
+
# IMAP
-
set maildir_trash = yes
+
set imap_authenticators="login"
-
set postponed="imaps://imap.gmail.com/[Gmail]/Drafts"
+
set imap_passive="no"
-
set header_cache = yes
+
set imap_user = 'username'
-
set smtp_url = smtp://username@gmail.com@smtp.gmail.com:587/
+
set imap_pass = 'password'
-
set smtp_pass = "password"
+
set spoolfile = imaps://imap.gmail.com:993/INBOX
-
</pre>
+
set folder = imaps://imap.gmail.com:993
 +
set record="imaps://imap.gmail.com/[Gmail]/Sent Mail"
 +
set maildir_trash = yes
 +
set postponed="imaps://imap.gmail.com/[Gmail]/Drafts"
 +
set header_cache = yes
 +
set smtp_url = smtp://username@gmail.com@smtp.gmail.com:587/
 +
set smtp_pass = "password"
 +
 
Replace username and password as needed. Launch mutt:
Replace username and password as needed. Launch mutt:
  mutt
  mutt
-
you'll be asked to confirm something about a certificate, then mutt will access your IMAP account and retrieve headers for all your emails. Yes, all headers (but not emails content...) ! There may be an option to only retrieve last email headers (only emails X days old), but I just don't know it. This can take quite a lot of time and kills your dataplan... Luckily this occurs only once: mutt is creating its header cache. When running again, mutt will just re-analyze its cache, locally. This can take quite some time anyway, when you have a large mailbox.
+
you'll be asked to confirm something about a certificate, then mutt will access your IMAP account and retrieve headers for all your emails. Yes, all headers (but not emails content...) ! There may be an option to only retrieve last email headers, but I just don't know it. This can take quite a lot of time and kills your dataplan...
-
 
+
-
Using mutt like this is unfortunately not that nice:
+
-
* your network connection must be fast, because browsing emails will constantly make network connection to the remote IMAP account
+
-
* worst case, when you don't have network connection, you just can't read your emails, even old ones...
+
== offlineimap to read emails when no network connection is available ==  
== offlineimap to read emails when no network connection is available ==  
-
While one major advantage to IMAP protocol is being able to directly and remotely access emails on an account, when you don't have any network connection, you just can't access them... A workaround is to use a tool like [http://wiki.github.com/jgoerzen/offlineimap/ OfflineIMAP], which can synchronize an IMAP account with a local maildir.
+
todo
-
 
+
-
This sync occurs both ways: changes on IMAP account will propagate to maildir, changes on maildir will propagate to IMAP account. This ensure what you'll locally read will also be marked as read on the IMAP server.
+
-
 
+
-
In order to use offlineimap, I had to get last sources including patches to select only emails not olders than X days. This will prevent getting *all* emails from the server (headers and bodies)... offlineimap package is available from extra-devel repository (optified to save some rootfs space). Install it as root:
+
-
 
+
-
sudo gainroot
+
-
apt-get install offlineimap
+
-
exit
+
-
 
+
-
Once installed, you'll need to create <code>~/.offlineimaprc</code> in order to define remote IMAP and local maildir accounts. Here's mine, again minimalist. See offlineimap.conf on OfflineIMAP website to figure out possible options.
+
-
<pre>
+
-
[general]
+
-
accounts = GMail
+
-
ui = Noninteractive.Basic
+
-
socktimeout = 300
+
-
 
+
-
[Account GMail]
+
-
localrepository = Local
+
-
remoterepository = Remote
+
-
maxage = 5
+
-
+
-
[Repository Local]
+
-
type = Maildir
+
-
localfolders = ~/Mail/GMail
+
-
sep = /
+
-
+
-
[Repository Remote]
+
-
type = Gmail
+
-
remoteuser = username
+
-
remotepass = password
+
-
realdelete = no
+
-
trashfolder = [Gmail].Trash
+
-
folderfilter = lambda foldername: foldername in ['INBOX', '[Gmail]/Brouillons','[Gmail]/Suivis']
+
-
nametrans = lambda foldername: foldername.replace("[Gmail]/","")
+
-
</pre>
+
-
We are basically telling offlineimap to sync a GMail IMAP account on a local maildir located in <code>~/Mail/GMail</code>. Only emails younger than 5 days will be considered. Only directories listed folderfilter will be synced. This is useful when you lots of GMail labels but don't want to sync every of them (this can take quite some time and bandwidth, depending your dataplan). Also, instead of having "[Gmail]/Suivis" directories, I prefer (and Modest too, see later) only having a "Suivis" directory. nametrans is used to translate local dirs, here we're telling it to remove the "[Gmail]/" part.
+
-
 
+
-
Also create maildir location:
+
-
 
+
-
mkdir ~/Mail/GMail
+
-
 
+
-
Finally, because my deb packaging isn't that good, let's create a wrapper bash script to run offlineimap:
+
-
<source lang="bash">
+
-
#!/bin/bash
+
-
offlineimap 2>&1 | tee /tmp/offlineimap.log
+
-
echo Last run: `/bin/date`
+
-
</source>
+
-
Save this script in <code>~/bin</code> (for instance) as "syncemail.sh", and make executable:
+
-
 
+
-
chmod +x syncemail.sh
+
-
 
+
-
Each time it's launched, you can check what it does looking as <code>/tmp/offlineimap.log</code>.
+
-
 
+
-
Time to test. Run it !
+
-
 
+
-
./bin/syncemail.sh
+
-
 
+
-
Once it's done, you can run mutt to check what it retrieved, telling to access a maildir:
+
-
 
+
-
mutt -f ~/Mail/GMail/INBOX
+
-
 
+
-
If it sounds good to you, you can program a task, for instance using Alarmed, and configure it to run this script every X minutes. If you don't plan or don't care having a GUI to setup this interval, you can also let offlineimap regularly check emails. In the account section, you can add something like:
+
-
 
+
-
[account GMail]
+
-
...
+
-
autorefresh = 15
+
-
 
+
-
this will tell offlineimap to sync every 15 minutes. Since it'll always run, you need to "daemonize" it when starting it. This upstart script can be saved as <code>/etc/event.d/offlineimap</code> to start it at boot time:
+
-
 
+
-
start on started hildon-desktop
+
-
stop on starting shutdown
+
-
service
+
-
console none
+
-
exec su user -c /home/user/bin/syncemail.sh
+
-
post-stop script
+
-
    kill `cat /home/user/.offlineimap/pid`
+
-
end script
+
-
 
+
-
CAREFUL: in <code>syncemail.sh</code> script, offlinemap will log into <code>/tmp/offlineimap.log</code>. Since we're daemonizing it, it'll run indefinitely. This log file may become huge. When fully tested, you should remove "2>&1 | tee /tmp/offlineimap.log" from syncemail.sh script. Also, post-stop step is here to actually kill the offlineimap python script, else only the bash launcher, syncemail.sh, will be killed, not the python process it runs.
+
-
 
+
-
So far so good, we have our emails synced. We can read them using mutt. Still, we may like to use a graphical front-end, fully integrated to N900 system (applet, widgets, contacts, etc...).
+
== Adding dovecot IMAP server to let Modest accessing emails locally ==
== Adding dovecot IMAP server to let Modest accessing emails locally ==
-
If only Modest could directly access our local maildir... Since it only support POP and IMAP account, we could install a local IMAP server, serving our maildir. Modest would then access this local IMAP server. This way, the remote sync part is delegated to offlineimap, while we can still use GUI front-ends. You'll still need to tell Modest to sync to the local maildir to check new emails. A short interval, 5 min, will ensure it'll be able to sync (since it's local, there's no network related issue anymore).
+
todo
-
 
+
-
So, there are many IMAP server out there, I found dovecot to be easy to package and use. dovecot can be found in extra-devel (also optified to save rootfs space). Install it as root
+
-
 
+
-
sudo gainroot
+
-
apt-get install dovecot
+
-
exit
+
-
 
+
-
Copy example configuration:
+
-
 
+
-
sudo gainroot
+
-
cp /usr/local/etc/dovecot-example.conf /usr/local/etc/dovecot.conf
+
-
 
+
-
Dovecot is very powerful, you can set a lot of parameters. I tried to keep it as simple as possible. Dovecot will <code>/etc/passwd</code> as it's user/password database. Here's my <code>dovecot.conf</code> file:
+
-
<pre>
+
-
base_dir = /var/run/dovecot/
+
-
protocols = imap
+
-
listen = 127.0.0.1
+
-
disable_plaintext_auth = no
+
-
ssl = no
+
-
login_user = dovecot
+
-
mail_location = maildir:~/Mail/GMail/INBOX
+
-
 
+
-
protocol imap {
+
-
}
+
-
 
+
-
auth default {
+
-
+
-
  mechanisms = plain
+
-
+
-
  passdb passwd {
+
-
  }
+
-
+
-
  userdb passwd {
+
-
  }
+
-
 
+
-
  user = root
+
-
}
+
-
</pre>
+
-
Note dovecot will serve INBOX only (more work/search to be done in order to serve multiple maildirs...). Since we'll use it locally, we can specify an IP address to listen. Else you (or someone else) could access your IMAP account using Wifi for instance.
+
-
 
+
-
You can check configuration file syntax by running:
+
-
 
+
-
dovecot -n
+
-
 
+
-
Dovecot auth will be done under user "dovecot". We need to create it:
+
-
 
+
-
sudo gainroot
+
-
useradd -g mail dovecot
+
-
 
+
-
Also, we will access maildir from Modest email client. This maildir is owned by user "user". This user doesn't have any password, but Modest won't allow you not specifying one. We thus need to set it (I'm not sure about potential side effects of setting a password for this default user. So far I haven't had any problems, but...):
+
-
 
+
-
sudo gainroot
+
-
passwd user
+
-
 
+
-
Launch dovecot, as root
+
-
 
+
-
sudo gainroot
+
-
/usr/local/sbin/dovecot
+
-
 
+
-
Test it's running with telnet:
+
-
 
+
-
telnet 127.0.0.1 143
+
-
 
+
-
should give something like:
+
-
 
+
-
* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE AUTH=PLAIN] Dovecot ready.
+
-
 
+
-
In order dovecot to be started at boot time, you can create this upstart script as /etc/event.d/dovecot
+
-
 
+
-
start on started hildon-desktop
+
-
stop on starting shutdown
+
-
service
+
-
console none
+
-
exec dovecot -F
+
-
 
+
-
Now open Modest, and add a new account:
+
-
 
+
-
Account name and name: whatever
+
-
username: user
+
-
password: yourpass
+
-
IMAP server: 127.0.0.1 # "localhost" won't be recognized as legal hostname...
+
-
IMAP: 143
+
-
No secure connection
+
-
 
+
-
About SMTP, you can use the one you're used to use. Another improvement would be to install a small local SMTP server, and let it send its emails when it can. So far SMTP sending works with Modest, so I didn't dig further.
+
-
 
+
-
== Extra: defining subcriptions for Modest ==
+
-
 
+
-
When running, dovecot will create a "subscriptions" file into the maildir (I think it's dovecot). You can edit this file to add more directories to display within Modest GUI. For instance, my <code>/home/user/Mail/GMail/subscriptions</code> is:
+
-
 
+
-
INBOX
+
-
Brouillons
+
-
Suivis
+
-
 
+
-
Note: if you don't translate GMail folder, that is, if you keep "[Gmail].Suivis" for instance, Modest won't display directory correctly and split it as "[Gmail]" and ".Suivis". That's why I added a nametrans paramater on my offlineimaprc configuration file.
+
-
 
+
-
 
+
-
== Conclusions ==
+
-
 
+
-
Using offlineimap and Dovecot, the synchronization process is removed from Modest email client. I'm still testing this configuration, it seems to work flawlessly. There are still issues and limitations:
+
-
 
+
-
* you can tell offlineimap to partially download emails, but you can't order it to fully download one specific emails (like on iPhone for instance). Attachments are also downloaded.
+
-
* some deb packaging would need to be improved. I did this quick & dirty
+
-
* you can't access different IMAP folders and different accounts. This would need to be improved, probably on dovecot side.
+
-
* "user" needs to have a password set. There can be unknown side effects. This could be avoid by setting a PAM authentication in dovecot (doesn't use /etc/passwd, no need to have a system user).
+
-
* it only deals with IMAP. Accessing POP account could be done the same way: instead of offlineimap, you could setup fetchmail or getmail and put retrieved emails in maildirs, just like here.
+
-
* ...
+
-
 
+
-
== Where to find packages ? ==
+
-
 
+
-
dovecot, muut, msmtp and offlineimap used to be available from the following links:
+
-
 
+
-
* [http://dl.dropbox.com/u/7090003/dovecot_1.2.11-1_armel.deb dovecot_1.2.11-1_armel.deb]
+
-
* [http://dl.dropbox.com/u/7090003/msmtp_1.4.20-1_armel.deb msmtp_1.4.20-1_armel.deb]
+
-
* [http://dl.dropbox.com/u/7090003/mutt_1.5.20-2_armel.deb mutt_1.5.20-2_armel.deb]
+
-
* [http://dl.dropbox.com/u/7090003/offlineimap_6.2.1_all.deb offlineimap_6.2.1_all.deb]
+
-
 
+
-
I've also created and uploaded deb packages for dovecot and offlineimap to extra-devel repository for convenience. You should prefer install them from extra-devel, as these packages are cleaner and optified (they install in /opt, not in / rootfs).
+
-
 
+
-
 
+
-
Hope it helps.
+
-
 
+
-
Cheers,
+
-
 
+
-
Seb
+
-
 
+
-
== modest & offlineimap "shortcut" ==
+
-
 
+
-
I am currently experimenting with this setup:
+
-
 
+
-
* offlineimap synchronizes normally (as above)
+
-
* the local repository looks like this
+
-
 
+
-
type = Maildir
+
-
localfolders = ~/.modest/local_folders/
+
-
 
+
-
* just to be safe, the remote repository does a nametrans: 'x' + foldername
+
-
* the end result is, that I can see synchronized folders in modest
+
-
 
+
-
Does anybody see a problem with this? It seems to work as it should.
+
-
 
+
-
* it seems that modest sees new mails as new only when you enter the folder, not in the folder overview (which would mean that in the long run, having built-in offline storage is better)
+
-
 
+
-
Gruss,
+
-
Christian
+
-
 
+
-
 
+
-
I don't know if it's the best place to comment but I tried to symlink ~/Mail/... with ~/.modest/local_folders/ and there were troubles (modest trying to create directory again ? I can't remember for sure). But this is interesting and could remove unnecessary complexity with dovecot. I'll try this setup !
+
-
 
+
-
Thanks & Cheers
+
-
Seb
+
-
 
+
-
[[Category:HowTo]]
+

Learn more about Contributing to the wiki.


Please note that all contributions to maemo.org wiki may be edited, altered, or removed by other contributors. If you do not want your writing to be edited mercilessly, then do not submit it here.
You are also promising us that you wrote this yourself, or copied it from a public domain or similar free resource (see maemo.org wiki:Copyrights for details). Do not submit copyrighted work without permission!


Cancel | Editing help (opens in new window)
Retrieved from "http://wiki.maemo.org/Improving_Modest_email_sync_reliability"