Original Request:

One thing about TB that always bothered me is that so much work is lost when
you need to reinstall from scratch because filters are so hard to back up, you
might want to consider a way of backing up and restoring filter rules.

Additional questions to answer

there are 2 use case scenarios: 
a) restoring the complete mail profile without changing anything
b) manual reinstallation on a new machine. In this case things like root folder
and account name are very likely to differ.


1 - on restore: should the existing filters be deleted or the restored filters
"Merged" into existing filters? In the case of "merge" how do we determine
whether it is a duplicate, and which one should we keep (existing filter or the
one from the backup)

2 - on restore: if the filters use "Tag as xyz" or "move to folder abc" as
actions, should they validate whether these folders / tags already exist? 

2a) What should they do if this validation fails?

especially in the case of folders there are a number of actions (with varying
difficulty)
- create the folder (based on the assumption that the target account is the
same as the source account, even if its name has changed)
- look for a similarly named folder and try to "transpose" the rule (this is
difficult to do because folder names are not unique, so there would have to be
a manual step which asks the user to confirm the "Matched" rules). The UI might
become complex.
- flag unmatched / non-functional filters via their name

(In reply to comment #1)
> Additional questions to answer
> 
> there are 2 use case scenarios: 
> a) restoring the complete mail profile without changing anything
> b) manual reinstallation on a new machine. In this case things like root folder
> and account name are very likely to differ.
> 
> 
> 1 - on restore: should the existing filters be deleted or the restored filters
> "Merged" into existing filters? In the case of "merge" how do we determine
> whether it is a duplicate, and which one should we keep (existing filter or the
> one from the backup)
> 
> 2 - on restore: if the filters use "Tag as xyz" or "move to folder abc" as
> actions, should they validate whether these folders / tags already exist? 
> 
> 2a) What should they do if this validation fails?
> 
> especially in the case of folders there are a number of actions (with varying
> difficulty)
> - create the folder (based on the assumption that the target account is the
> same as the source account, even if its name has changed)
> - look for a similarly named folder and try to "transpose" the rule (this is
> difficult to do because folder names are not unique, so there would have to be
> a manual step which asks the user to confirm the "Matched" rules). The UI might
> become complex.
> - flag unmatched / non-functional filters via their name
> 

I think the restore filters should be treated separately from the restore
profile, but as a component of it.
Import profile settings
 - import filters (checkbox)
 - anything else you were thinking of (another checkbox)
The idea of import as I see it requires to overwrite the previous filters, so
no merging, bad costly idea, but when you export, you can choose a file that
you add in a drop down menu for the import.
Look at Eclipse's import settings for an example:
http://help.eclipse.org/juno/topic/org.eclipse.platform.doc.user/tips/images/preference-transfer-compiler.png

As for 2, yes they should be validated and created if they don't exist (I want
to have the same folder tree as I have on the other computer, not to create
them manually - that's the point). Of course if the folder exists, with the
same name, at the same place, it won't be created (but the contents will be
overwritten).

I've thought more about 2. I understand now what you are saying. What if the
user created different folders on his other computer and now wants the mails to
be redirected to the new ones?

Current folders         Imported folders
folderA                 folderD
folderB                 folderE
folderC                 folderF

If the folders are different the user could opt out to: 
1) use the rules he has imported and keep all folders (the mails in A,B,C stay
there, and I will use the other folders from now on)
2) want folderA to be replaced by folderF (folderF is created and all mails
from folderA are moved to folderF and the rule is kept) - this is a bit more
difficult
3) don't want the new folders D,E and F, but the rules should use A,B,C

(In reply to comment #2)
> (In reply to comment #1)
> > Additional questions to answer
> > 
> > there are 2 use case scenarios: 
> > a) restoring the complete mail profile without changing anything
> > b) manual reinstallation on a new machine. In this case things like root folder
> > and account name are very likely to differ.
> > 
> I think the restore filters should be treated separately from the restore
> profile, but as a component of it.

This is out of scope for quickFilters. I don't want to program a full profile
restore, just import a list of filters. Also I am currently not interested in
even storing the quickFilters options as they are fairly trivial to set up.
Filters on the other hand can be numerous and complex.

> Import profile settings
>  - import filters (checkbox)
>  - anything else you were thinking of (another checkbox)
> The idea of import as I see it requires to overwrite the previous filters, so
> no merging, bad costly idea, but when you export, you can choose a file that
> you add in a drop down menu for the import.
Again, I am not sure that I agree.

If the user wants to start fresh it is in their digression to delete all
filters on the target account / mailbox before importing. it could however
delete all the filters that have the same name within a given mailbox.



> Look at Eclipse's import settings for an example:
> http://help.eclipse.org/juno/topic/org.eclipse.platform.doc.user/tips/images/preference-transfer-compiler.png

Looks a little complicated. I am not sure that I want to create a separate UI
that comes up when importing (your screenshot shows IMport prefs) without a
really important reason.



> As for 2, yes they should be validated and created if they don't exist (I want
> to have the same folder tree as I have on the other computer, not to create
> them manually - that's the point). Of course if the folder exists, with the
> same name, at the same place, it won't be created (but the contents will be
> overwritten).

Creating folder trees is not the point of quickFilters. Folders should be
either downloaded from IMAP or recreated by copying the Mail and ImapMail
folders. The filters should not generate folders.

(In reply to comment #4)
First you say:
> > > there are 2 use case scenarios: 
> > > a) restoring the complete mail profile without changing anything
Then you say
> This is out of scope for quickFilters. I don't want to program a full profile
> restore, just import a list of filters. 
Are you not contradicting yourself?

So you just import/export the filters, ok.

> > Import profile settings
> >  - import filters (checkbox)
> >  - anything else you were thinking of (another checkbox)
> > The idea of import as I see it requires to overwrite the previous filters, so
> > no merging, bad costly idea, but when you export, you can choose a file that
> > you add in a drop down menu for the import.
> Again, I am not sure that I agree.
> 
> If the user wants to start fresh it is in their digression to delete all
> filters on the target account / mailbox before importing. it could however
> delete all the filters that have the same name within a given mailbox.
I think this is not that much of a problem - you can either give the user a
choise (1. replace - wich deletes everything he has and writes the new ones or
2. overwrite - wich keeeps his original filters and overwrites the ones with
the same name) or notify the user that you will do the second one (overwrite).


> > Look at Eclipse's import settings for an example:
> > http://help.eclipse.org/juno/topic/org.eclipse.platform.doc.user/tips/images/preference-transfer-compiler.png
> 
> Looks a little complicated. I am not sure that I want to create a separate UI
> that comes up when importing (your screenshot shows IMport prefs) without a
> really important reason.
Ok, if you only need to just import/export the filters you don't need all that,
but what exactly is your idea, how do you want to achieve this?


> > As for 2, yes they should be validated and created if they don't exist (I want
> > to have the same folder tree as I have on the other computer, not to create
> > them manually - that's the point). Of course if the folder exists, with the
> > same name, at the same place, it won't be created (but the contents will be
> > overwritten).
> 
> Creating folder trees is not the point of quickFilters. Folders should be
> either downloaded from IMAP or recreated by copying the Mail and ImapMail
> folders. The filters should not generate folders.
This is probably case 1) from my comment #3. It will keep all folders (but you
should probably notify the user that this will happen).
If the user had filter rule 
- if "A" is contained in from then go to Folder A
and he imports
- if "A" is contained in from then go to Folder D
then probably all will go do Folder D after the import, so he should know this

(In reply to comment #5)
> (In reply to comment #4)
> First you say:
> > > > there are 2 use case scenarios: 
> > > > a) restoring the complete mail profile without changing anything
> Then you say
> > This is out of scope for quickFilters. I don't want to program a full profile
> > restore, just import a list of filters. 
> Are you not contradicting yourself?

No. I didn't say quickFilters is going to restore the mail. The use case is
valid for moving all mail to another machine or profile - and then wondering
"where are my filters gone"? Again, these file copy operations are not trivial,
so I am not going to create a full backup/restore utility (there is MozBackup
for this) when all the extension is about are filters. I would say the majority
of users at this stage use IMAP and have the folder structure sitting on a
server. In that case, the most important step is mapping the root folder to the
correct account. This means if you have 10 maili accounts the filters need to
be mapped to the correct ones when they are restored on the new system.



> 
> So you just import/export the filters, ok.
> 
> > > Import profile settings
> > >  - import filters (checkbox)
> > >  - anything else you were thinking of (another checkbox)
> > > The idea of import as I see it requires to overwrite the previous filters, so
> > > no merging, bad costly idea, but when you export, you can choose a file that
> > > you add in a drop down menu for the import.
> > Again, I am not sure that I agree.
> > 
> > If the user wants to start fresh it is in their digression to delete all
> > filters on the target account / mailbox before importing. it could however
> > delete all the filters that have the same name within a given mailbox.
> I think this is not that much of a problem - you can either give the user a
> choice 

> 1. replace - which deletes everything he has and writes the new ones 

TO be honest I am scared to do that - if the user chooses to delete all at
least it was their action if it goes wrong.


> 2. overwrite - which keeps his original filters and overwrites the ones with
> the same name) or notify the user that you will do the second one (overwrite).

sounds better - although currently the filters do not have date stamps so they
might overwrite updated filters with older versions, which might be a minor
inconvenience.


> 
> > > Look at Eclipse's import settings for an example:
> > > http://help.eclipse.org/juno/topic/org.eclipse.platform.doc.user/tips/images/preference-transfer-compiler.png
> > 
> > Looks a little complicated. I am not sure that I want to create a separate UI
> > that comes up when importing (your screenshot shows Import prefs) without a
> > really important reason.
> Ok, if you only need to just import/export the filters you don't need all that,
> but what exactly is your idea, how do you want to achieve this?

the idea is to add the missing filters that they are looking for. You idea on
name matching may be helpful (even though I cannot assume that the filter
hasn't changed in the meantime and we are actually overwriting good changes
with older bad ones)

> 
> 
> > > As for 2, yes they should be validated 
> > 
> > Creating folder trees is not the point of quickFilters. Folders should be
> > either downloaded from IMAP or recreated by copying the Mail and ImapMail
> > folders. The filters should not generate folders.
> This is probably case 1) from my comment #3. It will keep all folders (but you
> should probably notify the user that this will happen).

quote: "(the mails in A,B,C stay there, and I will use the other folders from
now on)"

which leaves the user with invalid rules - the point is remapping the target
folders which is the tricky bit:

how to find out that these should be remapped

Rule  => New Folder
D        A
E        B
F        C

the only thing we can know for sure during import is that the expected folders
D,E,F do not exist. I could develop a mapping algorithm that tries to trace
back from the child folder as many folders as possible in order to establish
with reasonable certainty that the new folder is D instead of A.

E.g.

Rule target URL:
mailbox://user@pop.vodafone.ie/Development/Mozilla/Addons/SmartTemplate/Support
1) search for all folders containing
"Development/Mozilla/Addons/SmartTemplate/Support" (URL minus account name) - 
  if matches are found assume 1st is a match
  no matches => continue with next step

2) search for all folders containing "Mozilla/Addons/SmartTemplate/Support"
(shortened URL minus account name)
  if matches are found assume 1st is a match
  no matches => continue with next step

..etc

5) search for all folders name "Support"
  if only 1 match is found => success

Each step is terminal but also if we have multiple matches then auto-mapping
might become problematic, so I need some UI for this or somehow mark these
filters as invalid - also if we cannot match the account with these, then I
need to carefully consider where to import them (maybe they could be imported
into Local Folder)

As you can see I still need to flesh out the algorithm some more but also, more
importantly, consider whether I can build a UI for the "Manual remapping" for
the filters that cannot be mapped to any folder on the system. 

One workaround could be not to import all filters for all accounts but let the
user pick the account first and then only backup / import the filters for that
account.

Probably the best, user friendly thing would be to present all folders to the
user and let him decide.

Local                     Import

ImapMail                  ImapMail
1  imap.googlemail.com    2   imap.something
2  imap.mail.yahoo.com    1   imap.something.else

And let him do the remapping somehow (like assigning a number to each one).
And then you can compare every msgFilterRules.dat by the rules he gave.

(In reply to comment #7)
> Probably the best, user friendly thing would be to present all folders to the
> user and let him decide.
> 
> Local                     Import
> 
> ImapMail                  ImapMail
> 1  imap.googlemail.com    2   imap.something
> 2  imap.mail.yahoo.com    1   imap.something.else
> 
> And let him do the remapping somehow (like assigning a number to each one).
> And then you can compare every msgFilterRules.dat by the rules he gave.
> 

Yes - listing all accounts is a good idea. I think the main decisions could
probably be made before the export - one "quick" way would be to have the
choice of having all filters in one file and conversely a choice of "one file
per mailbox" which could just be named after the account name. The file itself
can store whether it is is a "single account" or has multiple account rules.
Then when reading the file I will probably read in the raw data and give a
choice (if it contains multiple account rules).

So remapping the _source_ folder can be done manually that way. Still as I
often filter moving files across to another mailbox I need to gather all
possible account destinations and have them mapped as well in the way you
described.

 Examples from my own msgFilterRules.dat files:


actionValue="imap://user.name%40gmail.com@imap.googlemail.com/[QuickFolders]"
actionValue="imap://thunderbirddaily%40gmail.com@imap.gmail.com/[Gmail]/Social/Twitter"
actionValue="mailbox://user_name40hotmail.com@pop3.live.com/Inbox/SQLServer%20Central"
actionValue="mailbox://user@pop.vodafone.ie/Development/Mozilla/AMO/Zig%20Boom"
actionValue="mailbox://user@pop.iol.ie/Jobs/_Surpass/Gerard"
actionValue="exquilla://user@eumail.avaya.com/Inbox/Server%20Errors/BECS/Open%20Claims%20Reports"


From this, I could just gather a list of protocols+mailboxes and then let the
user choose which ones they should be mapped to, using multiple account
drop-downs.

After this a list of matching (& unmatched) folders for each account can be
generated. A percentage of "good" folder matches would also give some security
as to whether the imported rules will be successful. I wouldn't attempt to
"fix" all the bad folders but could potentially use the filter names to flag
invalid rules. E.g. by appending [it] (for invalid target) to the name. A
(semi) automatic fix routine for these could then be written as a phase 2.

I have assigned this feature as the next major feature to be added to
quickFilters pro (3.5) as a premium function. Adding the freemium model helps
pushing through difficult features like this.

(In reply to comment #9)
> I have assigned this feature as the next major feature to be added to
> quickFilters pro (3.5) as a premium function. Adding the freemium model helps
> pushing through difficult features like this.
> 

So a follow up - the premium version is released 
( see  quickfilters.mozdev.org/version.html#3.4.2 )

and also has sold a decent amount of licenses for Christmas, I am now actually
activaly working on the backup / restore feature. I still would like to be able
to merge in opld backups (to avoid losing newer filters) with the main problem
of filters not having a unique identifier (nor a date stamp) which means if a
filter is renamed in the meantime and I use the filter name as id then you will
likely end up with duplicates. 

Apart from that, overwriting the existing filters with their backed up versions
(regardless of their age) may be good behavior for a strictly backup / restore
scenario. For syncing, it may also be sufficient but it would be better if I
could figure out a way to save date stamps with each filter (as it is done with
passwords in firefox) so that we could give the users a choice of keeping more
recent edits of their filters especially when syncing filters across two
machines.

The date stamp mechanism in my QuickPasswords addon makes sure to only
overwrite entries with the same / older date stamp than in the backup file, so
that recently changed passwords are not affected.

Created an attachment (id=8356) [details]
3.5 prerelease 25

First test version, only SAVE works! load still needs to be implemented.

I decided to store the filters as JSON format for readability and not the very
convoluted way of msgFilterRules.dat

To Do: 
figure out a way of generating UID for each filter
force Thunderbird to store date stamps in its own storage system

changed priority to P1 so we can implement this feature for the next official
release of quickFilters

Created an attachment (id=8357) [details]
3.5 prerelease 38

Added some UI elements and logging ad improved file naming (added date stamp)

note: only English AND German Locale are complete!

Created an attachment (id=8371) [details]
3.5 prerelease 68

Added "Load" feature, first working version.

I didn't test quite a few cases, so use at your own risk. Definitely don't know
what is going to happen if a custom action is used. Old filters are currently
not deleted, and restored filters are added at the top. So you can delete the
original filters before import or go to the end of the "new" list and delete
the "remaining" items.

There is no "intelligent merge" or checking for existing filter names - filters
will simply be duplicated for now if you load the same list twice.

(In reply to comment #14)
> Created an attachment (id=8371) [details] [details]
> 3.5 prerelease 68
> 
> Added "Load" feature, first working version.
> 
Just a note: new filters are added at the TOP of the list, so existing filters
will be shifted down and executed last. If you keep old filters in order to
"merge" in a backup, make a note of the first existing filter name so you can
find it.

Hi not sure if anyone has covered this but have you considered making the file
format for the backup Gmail compatible as mailFilters.xml

Regards Dave

(In reply to comment #16)
> Hi not sure if anyone has covered this but have you considered making the file
> format for the backup Gmail compatible as mailFilters.xml
> 
> Regards Dave
> 

no, not so far. It may be a little difficult at this stage (if you expect
Thunderbird to read gmail filters that may be too much work...) but I am not
currently aware of this format. How do you create them?

Ideally I would like an option to sync across different accounts automatically
but I wouldn't like to restrict this to gmail accounts only. Which means it
would need a new mail filter standard or my own server (realistically about 3
man months of development + investment, so this is high risk for me and very
unlikely). AFAIK there is no syncable server supported filter standard outside
of Outlook / Exchange.

(In reply to comment #17)
> (In reply to comment #16)
> > Hi not sure if anyone has covered this but have you considered making the file
> > format for the backup Gmail compatible as mailFilters.xml
> > 
> > Regards Dave
> > 
> 
> no, not so far. It may be a little difficult at this stage (if you expect
> Thunderbird to read gmail filters that may be too much work...) but I am not
> currently aware of this format. How do you create them?
> 
> Ideally I would like an option to sync across different accounts automatically
> but I wouldn't like to restrict this to gmail accounts only. Which means it
> would need a new mail filter standard or my own server (realistically about 3
> man months of development + investment, so this is high risk for me and very
> unlikely). AFAIK there is no syncable server supported filter standard outside
> of Outlook / Exchange.
> 

Yeah in Gmail under settings filters there is export and import options.
Just a thought as you would need to change the format of the backup file from
the json to the xml and the import to xml rather than json so it works both in
your plugin and in gmail.

(In reply to comment #18)
> (In reply to comment #17)
> > (In reply to comment #16)
> > > Hi not sure if anyone has covered this but have you considered making the file
> > > format for the backup Gmail compatible as mailFilters.xml
> > > 
> > > Regards Dave
> > > 
> > 
> > no, not so far. It may be a little difficult at this stage (if you expect
> > Thunderbird to read gmail filters that may be too much work...) but I am not
> > currently aware of this format. How do you create them?
> > 
> > Ideally I would like an option to sync across different accounts automatically
> > but I wouldn't like to restrict this to gmail accounts only. Which means it
> > would need a new mail filter standard or my own server (realistically about 3
> > man months of development + investment, so this is high risk for me and very
> > unlikely). AFAIK there is no syncable server supported filter standard outside
> > of Outlook / Exchange.
> > 
> 
> Yeah in Gmail under settings filters there is export and import options.
> Just a thought as you would need to change the format of the backup file from
> the json to the xml and the import to xml rather than json so it works both in
> your plugin and in gmail.
> 

By the way is this the standard you are looking for?
https://en.wikipedia.org/wiki/Sieve_(mail_filtering_language)

Ìmplemented: Version 3.5 Released May 3, 2018