Storage in mail files make back-up difficult

I couldn't switch to IMAP even if I wanted to.

I must have access to my e-mail whenever I have access to my Mac without depending on access to the internet or my local network. E-mail just has to be permanently available no matter where I've taken my MacBook Pro to.

Apart from that keeping e-mail outside of my Mac would mean that I couldn't use Spotlight to search my e-mail and that would probably mean I might be spending more time searching for an e-mail than actually reading it. ( Delete or Edit)

@mikenolte - that's what offline usage is made for - you can decide (or should be able to do so) which folders you want to make available for offline usage.... then these folders are downloaded and cached at your local disk. ( Delete or Edit)

@Mike - Sorry, I should have pointed out that the three paragraphs are completely independent of each other.

First paragraph: I can't switch to IMAP simply because one of the servers I get mail from doesn't support IMAP and unfortunately that won't not change any time soon and I have no influence on this.

Second and third paragraphs: Pure online IMAP won't work for me but even if it did that wouldn't fix anything anyway because Postbox apparently would download and locally store all mail anyway.

There are two facts to focus on here:

Monolithic storage of e-mail is a bad idea for reasons stated above.

Even switching from POP to pure online IMAP doesn't get rid of local monolithic storage.

Using pure online IMAP without local caching might have been a viable workaround for some users but who says that these users' situation will never change? And then what?

My suggestion is: Let's just stop giving ourselves a pointless headache about how we might work around this if we could work around it, because there is no practical workaround anyway.

BTW: I talked to my lawyer today and he pointed out that IMAP might turn from a technical workaround to a legal problem. In Germany self-employed and businesses are required by law to keep all their business-relevant paperwork and e-mail for ten years. And to make it short: I'm not yet ready to trust my ISP to store and properly backup my stuff 100% safe for a decade. ( Delete or Edit)

@mikenolte - re Lawyer: there is great software that syncs IMAP servers to your local machine... benefit - if your local copy is destroyed (stolen, burned, whatever) there is a remote copy available... but data backup is another story.... ( Delete or Edit)

Actually it's not only a Time Machine problem.

The way Postbox stores e-mail in huge monolithic files is a gigantic problem for any serious back-up strategy no matter what operating system or back-up software people might be using. The tighter a back-up schedule is, the bigger the pain caused by Postbox will be.

Above I only mentioned the storage aspect but of course we all know that the data somehow has to get into storage. Let's have a look at the same example I used above to point out the incredible waste of storage Postbox would be responsible for.

28 People with an average 1.8 GB of of mail data would produce 50.4 GB of temporary Time Machine back-up data per hour. 50.4 GB/hour = 50400 MB/3600 seconds = 14 MB/sec. 14 MB/sec is more than your average 100 Mbit network's maximum bandwidth making 1 Gbit networks obligatory for Postbox in average small to medium size businesses and even then Postbox would eat up a significant amount of the total bandwidth permanently. And if you think this situation sounds a little crazy well then let me quickly remind you that I'm referring to real-life client of mine, they are not the biggest client I have and I am not exaggerating anything here.

Postbox will single-handedly blow your back-up storage strategy and your network bandwidth straight out of the water. ( Delete or Edit)

Things appear to be even are worse than that.

Although this topic is currently the most popular unsolved problem, there is not even a single official rep statement in this thread.

Go figure. ( Delete or Edit)

Time Machine cannot do block level back-ups. That's exactly why Postbox turns out to be a trouble maker. Actually I couldn't find a documentation for Jungle Disk or Mozy that explicity mentions block level back-ups are possible there. Where exactly did you get that information from?

Anyway, I'm ready to guess no Mac users in their right minds would ever dream of swapping Time Machine out for anything else because no other back-up software out there offers a one-stop solution for a complete system restore.

Let's say my hard drive dies on me suddenly and I replace it with a brand new black hard drive. Here's what I would have to do:
- Start my Mac using the Mac OS X 10.5 DVD.
- Optionally use disk utility to partition the new hard drive however I desire.
- Use Time Machine to restore my back-up data to the new hard drive.
- Reboot the Mac. Done.

Another back-up solution that can do this is yet to be found for Mac OS X. ( Delete or Edit)

Using block-level back-ups might slightly alleviate working with Postbox but it would still only be another work-around for a problem that urgently needs to be solved at the core.

An e-mail-client must not store e-mail in ever-growing monolithic files. Enough said.

Postbox will remain on glaobal ban lists as long as Postbox' developers choose to completely ignore this major problem. ( Delete or Edit)

I would appreciate a comment from the developers about why they chose to use mbox instead of .eml files - RFC-2822 message files. ( Delete or Edit)

While storing emails as individual files seems a brilliant idea, there is tha problem that each directory can store only a limited amount of files.
So in the long term, unless the mail client does not scatter the emails in several directories (creating madness sooner or later and making export and consolidation a pain in the ass) I am pretty sure that someone would come back here saying something like "I have a gazillion emails and suddenly postbox/thunderbird started losing the new ones. The messages are not saved on the disk anymore. Help, help..." and so on... ( Delete or Edit)

@gensuke72: The maximum number of files (or files and folders) in a folder (all Mac OS X versions) is 2^31, or 2,147,483,648. However, the actual number of files that can be stored on a Mac OS Extended (HFS Plus) or journaled HFS+ volume depends on the volume's size and the size of the files. For example, a 160 GB Mac OS Extended volume with the default block size of 4 KB has 40 million available blocks. This volume could store up to 40 million very small files, but not 2 billion. A bigger disk with the same default block size could hold proportionately more files. <http://support.apple.com/kb/HT2422>.

NTFS allows up to 4,294,967,295 or 2^32-1 files on the volume no matter in what folder they are. <http://technet.microsoft.com/en-us/li...>

If you truly believe that there is any problem with a maximum number of files per folder then please tell us what operating system and what file system you are talking about and name the official source of documentation for the limit you are referring to. ( Delete or Edit)

@Mikenolte:
I had problems with a Solaris box, but it was related to the maximum number of subdirectories, not files.

Now even if the filesystem allows the creation of a couple of billion files, I still remember clearly that managing a single directory with thousand of files was a big pain in the ass.

So maybe a nice way to handle the problem would be the creation of several subdirs in order to store single emails, something like year/month/day/emailfile00x.eml.

You would need a tool to eventually consolidate and export the emails towards other clients in case you need/want to go back to Mail, Outlook and he like.

Regards ( Delete or Edit)

@gensuke72:

Since there is no Postbox version for Solaris, we won't have to worry about that at all.

Don't waste your energy on too many variations of trying to compensate a problem you had way back when computers were still steam driven on a platform nobody can use to run Postbox on.

Whether or not manually managing a single directory with thousand of files was a big pain does not matter either because usually nobody will ever be manually working in that directory anyway.

> creation of several subdirs in order to store single emails, something like
> year/month/day/emailfile00x.eml

E-mail clients that store one file per e-mail exists for all platforms Postbox is running on and there are no problems related to having many files in the directories that store those e-mail files. It's not like we're discussing a hypothetical method here. One file per e-mail is simply the proper way to do it.

> You would need a tool to eventually consolidate and export the emails towards
> other clients in case you need/want to go back to Mail, Outlook and he like.

Not really a separate tool. Just the usual ability to export to mbox that any proper mail client will be able to import. Since Postbox already stores to mbox files that shouldn't be too hard to do. :-)

Postbox will need a proper export function anyway because creating a lock-in situation is another reason to get banned from corporate environments. ( Delete or Edit)

Actually they did "choose" this because they opted to just slap some eye-candy on Thunderbird and call it "development". Adding a method to properly store one file per e-mail would have been a lot less work than a couple of other things they introduced to Postbox. ( Delete or Edit)

I wouldn't wait for that to happen.

This was a problem in Thunderbird from the start and I'm ready to bet they knew about this even before they started thinking about Postbox.

They had their backs to the wall in Thunderbird and there's nothing new to say about Postbox.

Sticking their heads in the sands and quietly pretending there is no problem is all they can do. Postbox has already joined Thunderbird on corporate ban lists and apparently they simply don't care. ( Delete or Edit)

Just another thought - why do you need to backup Postbox' / Thunderbirds mail folder anyway? I hope you are using IMAP where your provider backs up the mail itself; the local data in this case is just a cache to speed up the operation, but does not hold all the mails itself... the idea behind IMAP.
So I'd just exclude the folder from the backup as its contents can be recreated any time.... ( Delete or Edit)

If you truly think that mbox format is the best way to store mails locally then you might be well advised to go and google the downsides of this format.

Concerning your demand for "all in one place": When each e-mail is stored in a single file then the folder that contains all the files is exactly *ONE* place. To have your e-mail in one place you don't need monolithic files.

You'd be mislead if you believed that Postbox stores all e-mail in one single monolithic file. It doesn't. It spreads your e-mail out into multiple monolithic files.

Your suggestion to use monthly files doesn't work for everybody either. Corporate e-mail usually isn't restricted to working with this month's and last month's e-mail. Changes do not only happen when new e-mail arrives. Frequent changes to your smaller monoliths would still bloat back-ups. And when a month's worth of e-mail might appear small enough for you, it still might be a huge pile of e-mail for somebody else. What will you suggest then? Weekly files? And while we're at it, daily files, too? Also we might as well dump the outdated counter-productive concept of monolithic storage for e-mail and keep only the method of using the one file per e-mail. ( Delete or Edit)

> I am also concerned the disk usage of one mail per file solutions.
> It's not very efficient to have thousands and thousands of files
> which size is couple of kilobytes (as usual plain text mails are).

If you have enough e-mail to worry about drive space loss due to thousands of files in your mail folder then you are already at a point where backing up your data has already reached a horribly painful state similar or worse to the case report I wrote about in my original post. The more e-mails you have the higher the probability that one file per e-mail we will be the mandatory method.

It is true that one file per e-mail will use slightly more disk space on the original data storage volume. But this amount of storage is tiny compared to the ever-growing huge amounts of extra back-up storage you will be needing in case you back-up monolithic e-mail storage.

> There is couple of good comments also about amount of files etc.
> ( FAT32 has limitation of 65,534 files in folder - and there are still peoples who use it)

Postbox requires at least Windows XP SP2 or Mac OS X 10.4. FAT32 is not an option for Mac OS X 10.4 users. And NTFS is default standard on Windows XP SP2. Who in his right mind would attempt to install Windows XP SP2 on a FAT32 file systems? FAT32's file number limitation will never be an issue for a Postbox user.

> Of course Scott and other gurus can solve those problems
> but it would take a lot of time and coffee.
> Is it worth it? I don't think so.

It's more than just "worth it", it's mandatory.

> Because Postbox is based on Thunderbird I think changing this fundamental behaviour
> could be difficult and pros and cons should be debated very thoroughly

Actually there won't be any discussion at all. The authors of Postbox have their heads firmly buried in the sand and corporation do not tend to discuss products that have already been hit by the big bad ban hammer.

> Is there another advantages with this one file per message approach than Time Machine?

Typically small and medium businesses won't have their own mail server and with monolithic storage problems as described above they will usually also not have proper daily back-ups of monolithic e-mail-storage even if their back-up plan is perfect in all other aspects. Data loss due to corruption of monolithic storage is common in corporate environments. Searching for "i lost my outlook database" gives you over 200,000 hits on Google. What more need I say? ( Delete or Edit)

mikenolte wrote: "Postbox requires at least Windows XP SP2 or Mac OS X 10.4. FAT32 is not an option for Mac OS X 10.4 users. And NTFS is default standard on Windows XP SP2. Who in his right mind would attempt to install Windows XP SP2 on a FAT32 file systems? FAT32's file number limitation will never be an issue for a Postbox user."

You can't say what users can and will do in their computer. There are many people who for example use FAT32 partition because they use same partition with Linux and Windows etc. Now there are proper NTFS implementation in Linux but still. And there are also users who has many OS's in their computer and FAT32 is their choice for compatibility reason. (I used to have both of this scenarios - Linux and Windows used same Thunderbird profile in FAT32 partition and one machine has Windows 98 for retrogaming and Windows XP for real use and shared FAT32 partition which contains documents and mail folders etc. It's dangerous you to say things like "FAT32's file number limitation will never be an issue for a Postbox user" or that your ways to think is the only way.
Maybe you should blame Apple and their retarded way to backup things...

Maybe changing current behaviour would be a good thing - but I don't think that's going to happen in near future. But it would be nice to have developers for example Scott's comment about it. I don't know Thunderbirds software architechture enough to comment about what kind of operations this change needs or is it possible at all. ( Delete or Edit)

> There are many people who for example use FAT32 partition

Might I bring back to your attention that all your ranting about how probable FAT32 usage might is quite pointless while we are still discussing a Mac OS X issue?

FAT32 is not an option for Mac OS X users. Mac OS X users do not have problems related to having many files in a single directory.

> Maybe you should blame Apple and their retarded way to backup things...

Well, maybe I should. But right now I'm quite happy with blaming Postbox for its even more retarded way to store e-mail in the first place. ( Delete or Edit)

GyazMail also stores one file per e-mail. GyazMail is available for Mac OS X only. ( Delete or Edit)

Another client that stores each message as an '.eml' file is Outspring Mail (OS X only). Comment: IMO Outspring Mail isn't ready for prime time, but it does show some promise. Unfortunately, the Outspring developers are far less responsive than the Postbox folks. ( Delete or Edit)

Any client that uses standard unix mail boxes even Microsoft Entourage. ( Delete or Edit)

Interesting point of view. Some one could reply to that you should say that Time Machine uses outdated model for saving snapshot. For example rsync doesn't have to save whole file when just part of it is changed. ( Delete or Edit)

Marko, if you were to implement a solution simply from the consistency and backup point of view - would you go for a single data file solution?
I could see this being a very good choice if you have limited capabilities in terms of available file handles, slow diskspace, bandwidth etc.
rsync is supposed to handle different types of data more on a server level in my opinion (fast syncs, reduce bandwith required etc.) Clearly a different focus if you ask me. ( Delete or Edit)

> It would be nice to compare Postbox when using one file per message

Yes, that would be really, really nice, because that would mean we'd have the option to store e-mail properly and we could stop worrying about this monolithic nonsense.

BTW: If you have a monolithic 500 MB e-mail database you'll propably be safe to bet on that taking longer to back-up then the changes of the last hour no matter how many different little files last hour's e-mail would stored in. ( Delete or Edit)

Marko, which OS are you using that automatically backups (and restores via e.g. drag and drop) email mailboxes via rsync. (Cron jobs, permissions, mounting etc.)?

"open file stream and close file stream etc.?" when was the last time you waited for a 200kb file to open? starting the (bloated) apps takes way longer. Also you are indexing usually on the spot and not hundreds of files once a day. Sorry we are not talking about mirroring server (-> rsync) but client app handling usually a few hundred kilobytes at a time.

But totally agree with the point regarding the storage format. Enough with the whole import/export nonsense. ( Delete or Edit)

I already answer to this sort of "recommendation" five months ago:

"I couldn't switch to IMAP even if I wanted to.

I must have access to my e-mail whenever I have access to my Mac without depending on access to the internet or my local network. E-mail just has to be permanently available no matter where I've taken my MacBook Pro to.

Apart from that keeping e-mail outside of my Mac would mean that I couldn't use Spotlight to search my e-mail and that would probably mean I might be spending more time searching for an e-mail than actually reading it. ( Delete )"

And just in case it managed to escape your attention: I do NOT use a Windows-PC. ( Delete or Edit)

So what you are saying that one should change the server because the client is "not working right" to handle the mailbox one has used for 10+ years? Absolutely sounds like the best solution. ( Delete or Edit)

I think you are passing on the point here. I use IMAP and I'm aware of the advantages. However, the way Postbox handles its mail database is an issue here for several reasons. BTW, I keep my mail as a local copy from the IMAP server so that I can access it and search it while I'm offline.

1) OS X users who use Time Machine (An incremental backup solution provided by apple and which is as simple as: add external HD, 2 clicks, you're set) and have a reasonable amount of email (lets say everything over a few 100 MBs) will see a dramatic increase in the time backups take and the size backups take because the whole .mbox file is backed up each time an email arrives.

2) With some technical knowledge, Time Machine can be configured to skip the mbox files. However, for IMAP and POP users who prefer to backup their local copy this is not preferred. (for example, because you dont want to download gigs of data over IMAP when your HD fails, restoring a mac with time machine takes an hour, restoring a mac including downloading several GB of mails takes multitudes of that.)

So, that puts a non-technical mac user in this position: "My backups are slow and I dont know why." This does not necessarily change their Postbox experience, but it does influence the overall experience of the OS.

And, a technical mac user will have to choose between two equally unpleasant alternatives. (1) Not using Time Machine to backup his email, or (2) Experiencing very slow and data intensive Time Machine backups.

That said, it's a problem, and the Postbox developers are in the position to change this for the best. (See post below) ( Delete or Edit)

Why don't you just symlink your Postbox folder to your second partition/drive? ( Delete or Edit)

Time machine backups hourly and makes a new backup if you receive or sent an email. In that backup all your emails will be copied. So, if you dont have that many emails in your email, you should be okay. If you do have a large email account, you should add the .mbox file that postbox uses to an exclusion list that time machine uses. (Open time machine preferences, click something with "choose folders not to backup" and go to ~/library/application support/postbox. Then exclude the profiles folder.

Do remember that you wont backup any settings from postbox this way. ( Delete or Edit)

One way you can keep the backed up mailbox small is that you don't let e.g. your inbox grow too much. For example move emails older than 30 days into an archive (trigger that every 30 days of course, not every day otherwise that file will be growing every day). ( Delete or Edit)

One way you can keep the backed up mailbox small is that you don't let e.g. your inbox grow too much. For example move emails older than 30 days into an archive (trigger that every 30 days of course, not every day otherwise that file will be growing every day). ( Delete or Edit)