Author Topic: Synch Settings  (Read 33693 times)

Offline tanstaafl

  • God Member
  • ******
  • Posts: 1363
Synch Settings
« on: September 22, 2005, 09:26:21 PM »
I'm posting a new Topic on this, because the other threads dealing with this Feature Request are confusing (they morphed from a different request).

Currently, there are these three requests:

1. Option to save settings to an FTP site

2. Import Settings into PasswordMaker Online

3. Merge Imports/Append Imports
(this last one actually spawned the first one, so they are both in the same thread)

What I'd like to propose is integrating the first two requests into one. I think the third one (merging) should be kept seperate for obvious reasons - basic Synch functionality should be much less problematic to implement.

Feature Request: Synch Settings

Detail:
Add the ability to 'Synchronize' PM settings to one (or more) FTP sites, designating one as the live 'Synch' site. The others would be used for backup purposes only (in case your Primary copy got trashed through a Server failure or something). By default, this could initially be the passwordmaker.org site, which would - with the proper backend coding - result in your Settings being available for use with the online version as well. Of course, the user would be more than welcome to use only their own servers for Synching if desired.

If Synching is not enabled, PM just operates the way it currently does.

The main issue I can think of is, what if you leave something running at home, then go to the office?

Well, IF PM can execute some code as a result of closing the last instance of FFox, then this should be addressable by making PM 'location aware' somehow (user chrome? a simple text file?). Each FFox browsing session would result in a Synch Session that has two parts - 'StartSynch', and 'StopSynch'. 'StartSynch' happens for the first instance of FFox opened, and 'StopSynch' happens when the last instance of FFox is closed. If a SynchSession is started, but never stopped, and another 'StartSynch' Session is attempted, PM would provide a warning with an explanation of the probable cause and a suggestion for how to deal with it.

I suggested this once before but Eric didn't seem to thrilled about the idea, but I really think that providing this kind of a service by passwordmaker.org should only be free for say  the first 30 (60? 90?) days, then if you wanted to continue to use passwordmaker.org's servers, you should pay a nominal fee. As Eric just pointed out, hosting and bandwidth aren't free. The cost could be nominal, but shoudl be enough to provide decent bandwidth to accomodate the service.
« Last Edit: September 22, 2005, 09:31:04 PM by tanstaafl »

Offline Eric H. Jung

  • grimholtz
  • Administrator
  • *****
  • Posts: 3353
Synch Settings
« Reply #1 on: September 24, 2005, 02:15:34 AM »
Hi tanstaafl,

As you mention, this is a great revenue opporuntity. I'm excited by that; the fundamentals and core of PasswordMaker will always remain free. But value adds are attractive to me and my small company, LeahScape :)   (yes, it is a real C corp, incorporated in Massachusetts)

Honestly, though, file synchronization is a notoriously difficult problem when there are n points of reference. A point of reference is an instance of a file on a given machine. Each point of reference has its own version history. When merging, every point of reference and every version must be taken into account. It's an extremely complex problem (at least for me). It requires much thought, much design, and an incredible amount of debugging. It also, in many cases, requires user-interaction (e.g., make one choice over another)

Imagine the headache I could cost people by releasing this feature with even the simplest of bugs, which, given the complexity of the problem, would be quite likely early on. People would be extremely "pissed off" that they're losing settings, and probably drop PasswordMaker like a hotcake.

Don't get me wrong--I understand the attractiveness of the feature because I myself am a PasswordMaker user and run into this synch problem every week.

I'm not saying it's out of the question. I'm just trying to set your expectations. Let's do this in stages: before we can even think about merging and synchronization, the fundamentals of FTP upload/download must be incorporated into PasswordMaker. Right now you can't even upload/downlolad passwordmaker.rdf to an arbitrary FTP site! The horse must come before the cart :)

I DO understand your request, your need. It is on the radar!

-Eric
« Last Edit: September 24, 2005, 02:18:08 AM by Eric H. Jung »

Offline tanstaafl

  • God Member
  • ******
  • Posts: 1363
Synch Settings
« Reply #2 on: September 26, 2005, 02:23:53 PM »
Hi Eric,

Quote
Honestly, though, file synchronization is a notoriously difficult problem when there are n points of reference. A point of reference is an instance of a file on a given machine. Each point of reference has its own version history. When merging, every point of reference and every version must be taken into account. It's an extremely complex problem (at least for me). It requires much thought, much design, and an incredible amount of debugging. It also, in many cases, requires user-interaction (e.g., make one choice over another)
I quite agree - that is precisely why I stated that *merging* two or more files should definitely wait until after basic Backup/Restore (what I called 'Synch') functionality is implemented... I realize now that the word 'Synch' does indeed suggest the merging of one file with another (is not what I meant), rather than simply replacing it (which is what I *did* mean) - my apologies for the confusion. :googly:

Quote
Imagine the headache I could cost people by releasing this feature with even the simplest of bugs, which, given the complexity of the problem, would be quite likely early on. People would be extremely "pissed off" that they're losing settings, and probably drop PasswordMaker like a hotcake.
Completely understood and agree...

Quote
Don't get me wrong--I understand the attractiveness of the feature because I myself am a PasswordMaker user and run into this synch problem every week.

<snip>

I DO understand your request, your need. It is on the radar!
Actually, my intent was to break it down even further - get LOCAL backup/restore of the RDF file working first - but I now see I need to be more clear. I have been working on the details, but they are not quite ready - I guess I should have waited until they were before I posted the updated request.

I'll be reworking this, complete with some separate posts with *suggested* methods for implementation the steps, and look forward to everyone else explaining why my way(s) won't work and providing ways that *will*.

Understand - my intent in providing implementation suggestions (for this Feature Request and any other I may make) is solely for clarity - to try to make it more clear what I am actually asking for, and to possibly give you a starting point on how it might be done - I am quite sure that you will come up with a much better way when you actually get to coding it, as you always seem to do... :)

Thanks for PM!

Offline quixin

  • Hero Member
  • *****
  • Posts: 538
Synch Settings
« Reply #3 on: September 26, 2005, 02:58:16 PM »
I think there are too many 'what ifs' involved with automated 'syncing'.  It needs to be all manual.  Like a split screen window with my current accounts on the left window and the remote accounts on the right window.

You can move accounts individually from one side to the other or move all at the same time.  It should never overwrite an account to avoid any pitfalls.  

What do you guys think?



Offline tanstaafl

  • God Member
  • ******
  • Posts: 1363
Synch Settings
« Reply #4 on: September 27, 2005, 02:35:04 PM »
Again, you are talking about *merging* the contents of two or more separate files.

My take is:

We should get simple *local* backup/restore capability of the *entire* *RDF* file working 100% first;
(this would allow me to easily perform new/clean installs and 'import' my settings upon first use (yes, I can do this manually, and I do, but doing thinsg like this manually is tiresome and error-prone)

*Then* introduce support for *multiple* locations/RDF files;
(this is much more tiresome and error-prone)

*Then* add support for *remote* backup/restore locations;
(ditto)

Then, and *only* then shoud we even *consider* implementing something as complex as 'merging' the contents of the RDF file(s).

Also, since the desired end result for probably 95% of the people using PM (I'm guessing here - I know I am one of them) is to keep the *same* settings across all of your browsers/profiles/computers, I don't even really see a major need for the complexity (and consequent error-prone nature) of file merging - but I concede the possibility that some people might want only *certain* accounts kept in synch, and not the entire file - but, since this is far more complex, and appeals to a much smaller audience, it only makes sense to get the basic code and UI working for complete file synchronizing done first.

Agreed?

Offline quixin

  • Hero Member
  • *****
  • Posts: 538
Synch Settings
« Reply #5 on: September 27, 2005, 02:44:49 PM »
I do agree that the basics come first.  The only reason I ever mentioned the ability to merge results, in truth from laziness.  

I use PasswordMaker at home and at work,  I never know when a new account is going to be created.  If I updated my home and work rdf file every time I made a new account it wouldn't be an issue.  The problem I find is having 3 or 4 accounts at home that are not at work, and 3 or 4 accounts at work that are not at home.  Maybe its uncalled for to request a feature that would allow me to fix a problem that could be avoided if I just kept up my rdf file both at home and at work on a regular basis.

To avoid having this problem I originally thought it would be a great idea to have PWM upload the RDF to an FTP site anytime there was a change or update.  Also to check the FTP for changes or updates every time it is started and apply the update if one exist.  That way it always staytes updates effortlessly.   Eric brought of the problem of having inconsistencies and what to do with them.  I figured manually merging was the best answer.  You can do the update whenever you wish and there would never be any problems because you would remain in charge.

Regardless, I do agree there are other essentials that need to be finished before this.

quixin



Offline tanstaafl

  • God Member
  • ******
  • Posts: 1363
Synch Settings
« Reply #6 on: September 27, 2005, 02:58:20 PM »
Quote
I do agree that the basics come first. The only reason I ever mentioned the ability to merge results, in truth from laziness.

lol! Isn't that why we use PM to begin with? ;)

Quote
To avoid having this problem I originally thought it would be a great idea to have PWM upload the RDF to an FTP site anytime there was a change or update. Also to check the FTP for changes or updates every time it is started and apply the update if one exist. That way it always staytes updates effortlessly. Eric brought of the problem of having inconsistencies and what to do with them.

Yes - this behavior is precisely what I describe in the request - I must have missed your original one...

The problem of having inconsistencies should be addressable by using date/time stamps, MD5 checksums, and a synch-session-status file of some sort on the Primary Synch Server. Describing the desired behavior is the easy part - the hard part will be Eric's job - the actual coding. ;)
« Last Edit: September 27, 2005, 02:58:41 PM by tanstaafl »

Offline ajw

  • Jr. Member
  • **
  • Posts: 81
Synch Settings
« Reply #7 on: October 05, 2005, 08:30:56 PM »
Having just started using PasswordMaker, I of course am an expert and know precisely how syncing should work...   (*huge* grin! :)

Actually, I've just read this thread and a few things do come to mind.

First, as you mentioned Eric, you must never lose data - a lost password would be a Very Bad Thing...

Let's take syncing between notebook and one other place (could be an FTP site, could be another computer - desktop at work or home, etc, could be a PDA.  Whatever...

Notebook is "A", other place is "B"

Activities on a record are:
    a new record is created
    a record is modified
    a record is deleted

So syncing has several cases:
1) a record is modified on A; B still has the old record
2) a record is modified on B; A still has the old record
3) a record is modified on both A and B
4) a record is deleted on A
5) a record is deleted on B
6) a new record is created on A
7) a new record is created on B
and combinations of these:
8) a record is modified on A, the same record is deleted on B
etc.


Because we never want to lose data, it's better to have two copies of a record than only one or the other.  This means that:
 - if a record is modified in more than one place, both records are retained
 - if a record is modified in one place and deleted in another, the modified record is retained (possibly with an indication it was deleted in the other)


Question:  how to tell chronology of changes?  timestamp is the obvious answer, but who's to say both places are in sync timewise - especially an FTP site; if A uploads changes to the FTP site, those records will have A's timestamp.  Later, B downloads but it's clock is off by a few minutes - couldn't B erroneously think an FTP record (modified by A) is newer or older, and thereby perform a wrong action?
Perhaps simply a record-modification-count would work.  Both sides would start a record at "1"; A modifies the record, so remembers it was 1 and is now 2.  B modifes record; same thing; was 1, now 2, and uploads to FTP site.   A syncs, sees that record is modified on both sides (record at last sync was 1 and both sides are now greater than 1, so modified on both sides).  Since it's modified on both sides, both records are retained (better to have both than to lose a modification)

Following that, each record is synchronized on both sides; if modified on one side, the other is updated.  If deleted on one side (and not modified on the other) it's deleted on the other.  If deleted on one side and modified on the other, the modified record is retained on both sides.  If modified on both sides, both records are kept (also on both sides)

Matching records from one side to another has to be something other than a simple index position; some kind of unique identifier kept in the record.
Multiple modifications of a single record (modified on both sides) could perhaps be the unique identifier and an additional duplication number.
So a record with unique identifier 12345-1 is modified on both sides, syncing would give two records; one with 12345-1 and the other with 12345-2
(this may introduce an anomaly with multiple devices synciing to the same FTP site; what if A syncs and has modified 12345-1.  then B syncs, also with a modified 12345-1.  B uploads its change as 12345-2.  Later, C syncs, also with a modified 12345-1.  Since there is already a -2, C uploads its record as 12345-3 (and downloads -1 and -2).  D does the same and creates a -4.   A syncs again, and downloads -2, -3 and -4 (it only has the -1 that it originally modified and synced; all the other duplicate mods happened after its first sync).

Then the user on A sees all 4 changes, merges them manually, and ends up with a newly-modified -1 and deletes -2, -3, and -4.   A syncs again, and updates the -1 on the FTP site (it's only changed on one side, so A overwrites FTP), and deletes -2, -3, -4.  B syncs later, downloads the modified -1 and deletes its -2, -3, -4.  As does C and D.

All databases now have the same history for record 12345.

Question:  Do deleted records have to hang around to differentiate between new creation of duplicates and deleted dups?   I.e., A has -1, -2, -3 after syncing with FTP1, then syncs with FTP2.   FTP2 only has -1.   Should A delete -2 and -3, or upload them to FTP2?   If FTP2 has a marker record saying -2 deleted, -3 deleted, A knows what to do.


This should allow syncing multiple machines to multiple "other places" (FTP sites, other machines, etc).

An enhancement could be that A syncs with FTP1, FTP2, FTP3 in sequence, and remembers if any changes are made after each FTP site; if so, goes and resyncs with the other FTP sites so at the end of the complete sync all the FTP sites are in sync with A.   (this could get messy if multiple machines are all syncing at once; A, B, C, D all syncing with FTP1, FTP2, FTP3, so ABCD are each changing all three FTP sites - while the others are syncing with them.  Eventually it'd all work out, but it could be a lot of re-syncs to do so.  But this is a really perverse case and I think pretty unlikely - it'd probably need all ABCD to be automatically syncing at timed intervals...)

A more typical case is A syncs with FTP1, then FTP2 and has some new changes, so it remembers it has to resync with FTP1.  Then syncs with FTP3 and has yet more changes, so has to resync with FTP1 and FTP2.  Goes back and syncs with FTP1 and FTP2 and it's done.


I know this trivializes the actual work - things like how to store duplicate records, how to relate them in database, UI changes to support duplicates, etc.  But I think this sums up syncing pretty well - or at least gets a lot of the points...

This also handles something like a PDA syncing with a desktop and the desktop syncing with an FTP site (one or more) - the PDA may lag behind (PDA syncs with A, A syncs with FTP1 and gets changes - PDA is behind until it syncs with A again)

BTW, I think PDA syncing is a dearly necessary thing - that's the reason I'm not using Roboform - they're syncing to the Palm is one-way; can't change data on the Palm side.  That's a lose for me.

Thoughts?  Ideas?  Flames?

- Al Weiner -

Offline tanstaafl

  • God Member
  • ******
  • Posts: 1363
Synch Settings
« Reply #8 on: October 05, 2005, 09:17:32 PM »
Hi Al,

Welcome to the forums! Thanks for your input. We've been discussing this for a while (there are some other threads on this subject elsewhere too), and are far from settling on evan a conceptual solution.

Quote
First, as you mentioned Eric, you must never lose data - a lost password would be a Very Bad Thing...
Well, technically, there are no passwords to lose - ;) - but of course, we all agree - loss or corruption of the Settings would be a 'Very Bad Thing', so reliability of this process is of utmost importance.

You have some good comments, and it will take some time for me to digest them, but I just wanted to make sure you understood one thing first...

What you are talking about is full 'synchronization'. The first stages of implementation will not involve synchronizing the changes between the files, it will be simple Backup/Restore functionality...

Hmm... reading your comments gave me an idea...

Maybe the best way to do this is to do the simple Backup/Restore capability first, then, for full blown synchronization, how does this sound...

1. Set up a real SQL Database on the PasswordMaker server.

2. Define a schema for the PasswordMaker Settings (lot of work there)... be sure to include a 'modification status' field (date/time stamp?) for every data field...

3. Make the local RDF file simply be a local storage container (XML file of course) for your Settings.

4. Write the Synch code, which will in effect perform basically the same, conceptually if not actually, as the Master/Slave functionality in OpenLDAP.

5. Build a web based interface that could then handle conflicts in some meaningfully intuitive way.

6. Fail-safe - provide a means for keeping a local backup copy, protected from the Synch Sessions, so that if somehow the data in the DB got hosed, you could just reupload your Settings from your last Backup - and make damn sure that a *validated* fresh Backup is made prior to any Synch Sessions.

This would give us *field* level control of modifications, in multiple directions and from multiple sources.

I'm not saying this would be *easy* of course - definitely a lot harder than simple FTP backup/restore capability - but it might be the best way to code a *reliable* method for accomplishing what we want that will work across all of the different implementations (PDA's coming immediately to mind as behaving differently than the Browser extensions)..

The only thing I don't like about it is it forces people to use the PasswordMaker Server if they want to be able to synchronize - at least at first. I guess we would make the DB schemas open as well, so if someone was capable of setting up their own DB server, they could host it themselves - and maybe it would become a common offering for Hosting Companies, along with Mailman and Apache.

Comments?

Offline ajw

  • Jr. Member
  • **
  • Posts: 81
Synch Settings
« Reply #9 on: October 06, 2005, 06:37:25 AM »
I'm sorry this is so long - I'm in verbal-spew mode these days...
I think I'm gonna find out if there's a message-lenght limit - and I'll capture the award for the longest posted message...


Hi Tan!  (if I may be so informal :)

I love your name!  I thought about using as a company name oh so many years ago - but then figured I'd have to explain it to potential clients and somehow it seemed like it could interfere with getting their business...   :)


Um, actually, the first three entries (uh, accounts) in my setup *do* have passwords; they're for credit cards and I'd already set up passwords for them and didn't want to go through the bother of changing the passwords on-line, and in FireFox, and in my PC/Palm password app...   So losing those *would* be bad for me.

(I'm hoping that PasswordMaker can replace the PC/Palm app I'm using now; it works fine, but it doesn't fill in web forms so I'm looking to replace it.  I know PM doesn't have a Palm app yet, but I'll wait a while for it.   I'd take a stab at it myself (I write Palm apps anyway) but I don't know much about Java and even less about Java on Palm devices - my background is embedded code and device drivers; gimme low-level C and assembly language and let me code to the metal!  :)


Same thing - it'll take me time to digest your response too, but at first peek...


Simple backup/restore is easy - just make a big blob of the data...  ("easy" is relative of course - as a friend says "Just a simple matter of programming..."  :)

Ok, you've got that - Export/Import  (although I've only tried Export; I'll take it as a given I can Import correctly too...)
So now you need a "copy this to the FTP server" - with appropriate security for transmission (HTTPS?  SSL?  I dunno - I don't know this stuff beyond alphabet soup)
and storage.


Before you go off into the SQL database (not a bad idea, I think - I played with MySQL and had a ton of fun learning, although I never followed up on my own project with it - wouldn't mind having a reason to do so again though, but you don't want a newbie designing the schema - or at least I *hope* you don't!  :)

I think the synchronization should support machine-to-machine syncing - that's what PDAs do.  They won't sync to the FTP server.


Quote
What you are talking about is full 'synchronization'. The first stages of implementation will not involve synchronizing the changes between the files, it will be simple Backup/Restore functionality...

Ok, but you've got that locally with RDF files.  Why not at least do the remote backup (FTP site) record-by-record - that leads into full syncing.   Otherwise you're pretty much going to throw away the backup-to-FTP code.   (except that users could set up their own FTP site, so it wouldn't be a bad feature to have anyway)

Throw in the fields you think you'd need for full-sync - modification date, modification status (modified/deleted/created/unchanged), duplication id (my -1, -2...) and so forth - fill 'em in as expected for a new record and then you'll have 'em for existing customer data when full-sync goes live.  If it turns out those fields are totally dumb-as-bricks then you're no worse off than if they weren't there in the first place, so it's only a win, except for the additional disk space.

Unless FTP backup to user's own FTP sites is planned, I think the additional work of backup record-by-record instead of one big file isn't that much additional work - of course I'm not coding it...   :)


Quote
...SQL Database...
I think this is a great idea - why reinvent the wheel?


Quote
2. Define a schema for the PasswordMaker Settings (lot of work there)... be sure to include a 'modification status' field (date/time stamp?) for every data field...
I'm not sure a date/time stamp is ideal.

What if I want to sync with my ISP's PasswordMaker Backup Server - and also with yours (and maybe 8 or ten others just cause I'm making the example difficult...)

What if a clock is off?  Or more likely, modifications occur on different machines, and are sync'd to only some of the backup servers.   Now you've got a mess, 'cause my ISP's server says the data for my Yahoo backup changed (by my notebook) at 9am, but later at work I changed my desktop and only sync'd with your server so your server says the record changed at 10:30.   But the data isn't the same - you need to keep both (what I called duplication number - maybe not the best name for it; maybe "modification number" or something...)

I think the description I gave with 1345-x (and having -1, -2, -3...) duplicatation records would work in this case.  If my mental meandering is right, even if a record (or its duplicates) are changed on multiple clients (notebook, PDA, work, home, etc) are synced with only *some* servers (so servers themselves are out-of-sync with each other) and it's made as messy as possible - think about the -1, -2, -3... records being created as I described, by client changes in several places and then syncing, but add in that some clients are only syncing with some servers (maybe I can't get to my ISP's server from work, for example) so we end up with even the -1, -2, -3 records being modified multiple times.  Deleted on some servers by some clients even as other clients have modified them.

I think it'll still end up right once all the syncing stabilizes - effectively after one client syncs with all the servers (and resyncs with servers as necessary, so all the servers are again in sync) and then all the clients sync with any of the servers.

We'll end up with additional duplication records (-5, -6, -7, etc) as the duplication records are modified - and as the clients create "collisions" of duplication records; two clients both modify -2, for example, and then sync to different servers - we now have -2 records on different servers with different data.

But when a client syncs with the first server (with the modified -2), it'll see that -2 is different than its own, so will change its -2 to -5 and download the server's -2 (and uploads its former -2; now -5).

Then it syncs with the other server (with the second modified -2) and it'll again see that its (the client's) record is different, so it changes its local -2 to -5 (it hasn't seen the server's -5 yet) and downloads the server's -2.

When the client sees that the server has a -5, and it's different from its own, it changes its (client's) -5 to -6, downloads the server's -5 and uploads -6.  (unless the server has a -6, in which case this happens again; client -6 becomes -7...)

If I'm right, multiple clients changing the same records and syncing with multiple servers will all end up with the same data.


If this is done with timestamps, I think either you'll end up erroneouosly thinking records are newer or older, or you'll think records are different when they're the same, and create spurious duplicates.   (what if I change a record at home from X to Y and FTP sync, then change the same record at work from X to Y and FTP sync - will I end up with one or two records?   Should be one record, I think; the data's the same, and that's what counts.

I may be wrong - timestamps might work just as well as my duplication number, but I've got a nagging feeling about it...


Quote
3. Make the local RDF file simply be a local storage container (XML file of course) for your Settings.
I'm not qualified to comment on this - but that's never stopped me!
Uh, whatever works...   :)


Quote
4. Write the Synch code, which will in effect perform basically the same, conceptually if not actually, as the Master/Slave functionality in OpenLDAP.
Again, not qualified, so I'll take your word on it.


Quote
5. Build a web based interface that could then handle conflicts in some meaningfully intuitive way.
That'd be nice, but is it necessary?  I think it'd be better to have conflict resolution handled in the client.  Ideally, the user would see the their original record and duplicates side-by-side and be able to merge as desired, but at first even a manual merge jumping back-and-forth between the records would be ok.
No, scratch that - that'd suck.  But it'd be ok during server design...
(Oh, it's *so* easy to make work for other people!  :)


Quote
6. Fail-safe - provide a means for keeping a local backup copy, protected from the Synch Sessions, so that if somehow the data in the DB got hosed, you could just reupload your Settings from your last Backup - and make damn sure that a *validated* fresh Backup is made prior to any Synch Sessions.
Yes!  Absolutely!  Might even keep a duplicate of the backup on the server - belt-and-suspenders...
Would be cool to be able to roll back the sync - both locally and on the server, but I think that's a nest of worms that make this whole discussion look like a cakewalk.  (what if another client syncs before the first client requests a roll-back...)
But a local pre-sync backup is a great idea.  Perhaps even keep several aged backups.   (lest a user sync with one server, then another, and another - and *then* discover he's hosed)


Quote
This would give us *field* level control of modifications, in multiple directions and from multiple sources.
If I read you right, that would allow multiple modifications to be shown in the single record, with multiple data.   So instead of two "credit card foo" records (-1, -2 in my nomenclature) I'd see a single "credit card foo" record, and something like:

Name:  Credit card foo Visa
changed on 10/4/05, was:  foo card

Username:  alan
changed on 10/6/05, was:  alanjay
changed on 10/4/05, was:  alan123


That would be incredibly cool - and way better than my duplication records!

Display of change history would be optional, of course.
Should the user be able to remove the change history?   Probably, but I wouldn't care if they couldn't - it's a valuable record, so I don't mind if it's permanant.
(would be on the server, anyway - or should they be able to delete the history there too?)

Record deletions might be tricky - delete on A, sync, modify on B, sync...   You'd have to see something like:

Record deleted on 10/1/05 (by client X)
Record restored due to modifications on 10/2/05 (by client Y)


Maybe that history should include all the modifications (optional?) so it'd show:

Record deleted on 10/1/05 (by client X)
Record restored due to modifications on 10/2/05 (by client Y)
Name changed on 10/4/05:    was:  foo card
                                 changed to:  Credit card foo Visa
Username changed on 10/4/05:  was:  alan123
                                     changed to:  alanjay
Username changed on 10/6/05:  was:  alanjay
                                     changed to:  alan123

Uh, I showed the changes in the single-record display above in descending date order, here I showed in ascending date order - probably should be user-configurable...  (I'm big on user options - especially when someone else is coding... :)


Quote
I'm not saying this would be *easy* of course - definitely a lot harder than simple FTP backup/restore capability - but it might be the best way to code a *reliable* method for accomplishing what we want that will work across all of the different implementations (PDA's coming immediately to mind as behaving differently than the Browser extensions)..
Would PDAs be syncing to the FTP server?  I was figuring they'd sync locally, like Palm or PocketPC does now...


Quote
The only thing I don't like about it is it forces people to use the PasswordMaker Server if they want to be able to synchronize - at least at first. I guess we would make the DB schemas open as well, so if someone was capable of setting up their own DB server, they could host it themselves - and maybe it would become a common offering for Hosting Companies, along with Mailman and Apache.
Could create a standalone module that contains a web server and DB server (SQLite?) - that would simplify things for end-users who wanted to run their own server.  Roundup bug-tracking system does that; I think Trac does too.  So it's doable; not sure if the work is worth it though, it *is* easy enough to set up Apache and MySQL.  Well,...  a single standalone app *would* be easier...


Quote
Comments?

Uh, I think I provided a plethora - aren't you sorry you asked?

- Al -

Offline tanstaafl

  • God Member
  • ******
  • Posts: 1363
Synch Settings
« Reply #10 on: October 06, 2005, 01:18:15 PM »
Quote
Hi Tan! (if I may be so informal smile.gif
Hey, I've been called a lot worse...

Quote
Um, actually, the first three entries (uh, accounts) in my setup *do* have passwords; they're for credit cards and I'd already set up passwords for them and didn't want to go through the bother of changing the passwords on-line, and in FireFox, and in my PC/Palm password app... So losing those *would* be bad for me.
lol - haven't used that tip yet, so wasn't thinking about it... but in all seriousness, you already *had* those passwords - you just told PM about them - so could you then really blame PM for losing something that it wasn't responsible for creating in the first place could you?

Quote
(I'm hoping that PasswordMaker can replace the PC/Palm app I'm using now; it works fine, but it doesn't fill in web forms so I'm looking to replace it. I know PM doesn't have a Palm app yet, but I'll wait a while for it.
There're a lot of features on the Table. I've thought about asking Eric to provide access to his secret 'priority' list of enhancements - the order he is actually planning on working on them - but I don't want to put him on the spot or anything. It would be nice though if there were some kind of target date for the feature list - not anything that could be relied heavily on, but you know - 'is my Feature a month away, or a year away' kind of thing... but keeping up with that is just one more thing that he would have to do, and I for one do not want to burden him with anything that isn't absolutely necessary...

Quote
So now you need a "copy this to the FTP server" - with appropriate security for transmission (HTTPS? SSL? I dunno - I don't know this stuff beyond alphabet soup) and storage.
Yes, this has been discussed as the first step...

Quote
I think the synchronization should support machine-to-machine syncing - that's what PDAs do. They won't sync to the FTP server.
I understand that you are saying this with the idea of the PDA in mind, and I agree that this may be the best way to go for the PDA version of syncing - it would only sync with a desktop version, at least initially.

However, the ubiquitousness of wirelss devices like PDAs (and more importanly, PDA phones like the Treo) I think will demand that they ultimately be able to Sync directly with the Primary (or one of the Slave) Sync Server(s).

Quote
Why not at least do the remote backup (FTP site) record-by-record - that leads into full syncing. Otherwise you're pretty much going to throw away the backup-to-FTP code. (except that users could set up their own FTP site, so it wouldn't be a bad feature to have anyway)
2 things...

First, the simple FTP Backup Restore functionality should always be there - many people won't want the complexity of full sync, and just want their settings to follow them from their home computer to their work computer. Secure FTP backup/restore does that just fine, with much less room for corruption.

Second, the GUI that will have to be designed to allow the User to work with multiple Files across multiple locations for the FTP Backup/Restore will most certainly still be usable for Full Synching, although it will have to be expanded on. So, not only will the code for plain FTP support remain, designing the GUI for it will provide the base layout for the advanced GUI needed for full Sync - at least, thats my take on it - but IANAP, so could be totally off base here...

Understand, its not that I'm against going ahead with the Full Sync stuff - its just that I don't really need that at this point, and I badly need a simple way to have my settings follow me around without having to manually email my RDF file back and forth (and forgetting to do this, and then being stuck with having to remember any new Accounts I added at the other end)... bah, I just want to be able to tell PM to upload my RDF file when I quit FFox, and then tell it to auto d/l it when I startup. I'm lazy... ;)

Quote
Unless FTP backup to user's own FTP sites is planned, I think the additional work of backup record-by-record instead of one big file isn't that much additional work - of course I'm not coding it... smile.gif
It is absolutely planned. The main reason for using passwordmaker.org was to also keep your Settings in sync with the online version hosted there (and at the same time providing a small revenue stream for Eric). But no one would be *forced* to use passwordmaker.org - all they would need is secure FTP server access.

Quote
Quote
2. Define a schema for the PasswordMaker Settings (lot of work there)... be sure to include a 'modification status' field (date/time stamp?) for every data field...
I'm not sure a date/time stamp is ideal.
Hence the ? after it. I was indicating that it wouldn't be sufficient in and of itself, but wasn't sure what other methods could be employed. It should be there, though - for those of us who keep our systems time-synced, we could then rely on this info - or at least it would help give us an idea of when changes were made...

Quote
What if I want to sync with my ISP's PasswordMaker Backup Server - and also with yours (and maybe 8 or ten others just cause I'm making the example difficult...)
These examples need to be difficult - we need to consider the 'worst case scenario' and plan for that.

Personally, I think the scheme should require one of the Sync Servers to be designated as the 'Master' - all others would be secondary. A totally distributed Multiple Master setup just isn't really necessary, although...

Hmmm... if this (Full Synchronization) was implemented strictly as an LDAP schema/solution, as opposed to an SQL DB, and required LDAP Servers for the Sync Servers, then however you set up your LDAP servers is how your Sync would work.

Anyone remember the days of the LDAP based Roaming Profiles in Netscape 4? I never used one, but my understanding is they worked very well, when implemented properly.

Even better, passwordmaker.org could be the first Master OpenLDAP Server. Make the schemas freely available, provide a process for LDAP guru's to help make the schema better - how long would it take before PAsswordMaker LDAP Mirror/Multiple Master Servers started springing up everywhere? It could act as the *only* (Master) server for those Users who didn't need the bells and whistles, but for those who did, they would be free to implement it any way they wanted that was supported by the LDAP schema.

Quote
I think the description I gave with 1345-x (and having -1, -2, -3...)
snip

I pretty much agree with all of your discussion on why time stamps won't be enough by themselves - but since I never intended that... ;)

Quote
(Oh, it's *so* easy to make work for other people! smile.gif
You got that right. It always takes me a long time to write up a Feature Request, because I'm trying to figure out the easiest way to accomplish what I'm asking for. My biggest problem, though, is my requests are often so complicated there is no easy way to do it...

Quote
Would be cool to be able to roll back the sync - both locally and on the server, but I think that's a nest of worms that make this whole discussion look like a cakewalk.
I disagree. You should be able to rely on the Date/Time stamps on the local copies to determine when they were made - and same for the servers. The potential problem would only be when comparing one on the server to a local one (or vice-versa).

I do think the User should be able to define how many local emergency backup copies PM archives - it's not like these RDF files are that big or anything.

Quote
Quote
This would give us *field* level control of modifications, in multiple directions and from multiple sources.
If I read you right, that would allow multiple modifications to be shown in the single record, with multiple data. So instead of two "credit card foo" records (-1, -2 in my nomenclature) I'd see a single "credit card foo" record, and something like:

Name: Credit card foo Visa
changed on 10/4/05, was: foo card

Username: alan
changed on 10/6/05, was: alanjay
changed on 10/4/05, was: alan123

That would be incredibly cool - and way better than my duplication records!
Yes, but as you said - it's easy to make work for someone else - I have a feeling this kind of thing would be incredibly complicated to code. That's why I'm trying to think of ways to take advantage of some kind of Client/Server solution that already has methods in place for synchronizing changes between multiple locations - and the more I think about it, the more I like the idea of using OpenLDAP for this. The hard part will be developing the schema, and then writing the interface (and I believe there are a couple of decent Open Source LDAP tools that we could probably hijack to get a jump-start) - but there are probably a lot of LDAP guru's out there who would ne happy to help, if we could get them hooked on PasswordMaker (which wouldn't be too hard to do - for the most part, all we should have to do is get them to try it)...

Quote
Display of change history would be optional, of course.
Should the user be able to remove the change history? Probably, but I wouldn't care if they couldn't - it's a valuable record, so I don't mind if it's permanant.
(would be on the server, anyway - or should they be able to delete the history there too?)
I like User options for things like this. The OpenLDAP Admin should be able to force cleanups based on aging or something, but other than that, let the User decide how much or little to keep.

Quote
Maybe that history should include all the modifications (optional?) so it'd show:

Record deleted on 10/1/05 (by client X)
Record restored due to modifications on 10/2/05 (by client Y)
Name changed on 10/4/05: was: foo card
changed to: Credit card foo Visa
Username changed on 10/4/05: was: alan123
changed to: alanjay
Username changed on 10/6/05: was: alanjay
changed to: alan123

This would definitely require an SQL option. Doable, maybe, but definitely the 5th or 10th version of Sync.

Quote
Would PDAs be syncing to the FTP server? I was figuring they'd sync locally, like Palm or PocketPC does now...
Definitely at least at first. As I said before though, with wireless always on connections, it makes sense to at least plan for the day when they can sync directly to a server.

Quote
Could create a standalone module that contains a web server and DB server (SQLite?) - that would simplify things for end-users who wanted to run their own server. Roundup bug-tracking system does that; I think Trac does too. So it's doable; not sure if the work is worth it though, it *is* easy enough to set up Apache and MySQL. Well,... a single standalone app *would* be easier...
True, and a good idea - *if* this standalone server can communicate with an LDAP server. I don't see any reason it can't, but again, IANAP, so...

Quote
Quote
Comments?
Uh, I think I provided a plethora - aren't you sorry you asked?
Not at all... discussions like this are needed to flesh out the best way to apporoach the problem. I'm sure Eric has done a lot of thinking on his own, but maybe we can give him some ideas that he might not have come up with otherwise.

Thanks for your input!

Offline ajw

  • Jr. Member
  • **
  • Posts: 81
Synch Settings
« Reply #11 on: October 09, 2005, 09:23:29 AM »
finally getting around to responding back to this...


Quote
Quote
Um, actually, the first three entries (uh, accounts) in my setup *do* have passwords; they're for credit cards and I'd already set up passwords for them and didn't want to go through the bother of changing the passwords on-line, and in FireFox, and in my PC/Palm password app... So losing those *would* be bad for me.

lol - haven't used that tip yet, so wasn't thinking about it... but in all seriousness, you already *had* those passwords - you just told PM about them - so could you then really blame PM for losing something that it wasn't responsible for creating in the first place could you?
*Absolutely* - if PM is going to be more than a "simple" password generator.  If it'll be a place I put all my secret information then it darn well better keep it all safe and be reliable in doing so!  Just imagine the horror of needing those "launch the nukes" codes, and your darn password generator lost 'em 'cause "I didn't make it so it's not important if I lose it..."   :)

Hmmm....   maybe there *are* times to lose that information...

And (I argue) if PM isn't going to store all my secret info, then I need another app to keep them in - and why would I want multiple places for secret stuff?  If I've gotta have a Master Secret Repository, then I'll just keep my passwords in that too.

It goes back to wanting *one* place to keep things - with duplication on my PDA 'cause that's always with me (but it can't fill out the web site when I'm on the notebook, so it needs to be on the notebook too...)


Quote
There're a lot of features on the Table. ...
Yeah, I agree - don't say when something'll be done 'cause then people get upset if it's "late".  No desire to burden Eric with more organizational junk - that's the stuff I hate myself, so I don't wish it on others!  :)
Much better to be coding than juggling lists of what to do...



Quote
Quote
I think the synchronization should support machine-to-machine syncing - that's what PDAs do. They won't sync to the FTP server.
I understand that you are saying this with the idea of the PDA in mind, and I agree that this may be the best way to go for the PDA version of syncing - it would only sync with a desktop version, at least initially.

However, the ubiquitousness of wirelss devices like PDAs (and more importanly, PDA phones like the Treo) I think will demand that they ultimately be able to Sync directly with the Primary (or one of the Slave) Sync Server(s).
Good point with the wireless movement - I've got a Palm Tungsten C (amongst other devices - way too many here...)  It's got 802.11b - wireless syncing is very cool!  And it could as easily sync to an FTP site as the desktop...

That could also be a revenue stream for Eric - syncing over cellphone.  Carriers are always looking for bandwidth-abusing apps.   (maybe it'll need to sync 15 times and compare all the copies just to be sure it's right - just to use up enough bandwidth to make the carriers like it...  :)

Still, at least in the near term, PDA-to-desktop is still reasonable - but probably should be done as a standard Palm conduit (or its ActiveSync equivalent).

Quote
Quote
What if I want to sync with my ISP's PasswordMaker Backup Server - and also with yours (and maybe 8 or ten others just cause I'm making the example difficult...)

These examples need to be difficult - we need to consider the 'worst case scenario' and plan for that.
Oh, goodie - I'm great at complexifying things!  :)

I don't agree on the master server though - I don't think it's necessary, and if there's a boss machine, what if that one's not available for some reason?  If it's all peer-to-peer, then one can drop out and it's no big deal.

Quote
LDAP...
I don't know much (read anything except a quick skim through RFC 1823 (and something else which I've already misplaced...  :)

Would LDAP sync a user's complete information in one blob?  Or record-by-record?  (PM account-by-account)  I got the impression that LDAP was unable to have information linked to two places - like an account under two groups (that's planned soon, right?)  Would LDAP handle that?   Or am I all wet and should go read more?  (ok, that's true regardless...  :)

Quote
Quote
Quote
This would give us *field* level control of modifications, in multiple directions and from multiple sources.

If I read you right, that would allow multiple modifications to be shown in the single record, with multiple data. So instead of two "credit card foo" records (-1, -2 in my nomenclature) I'd see a single "credit card foo" record, and something like:

Name: Credit card foo Visa
changed on 10/4/05, was: foo card

Username: alan
changed on 10/6/05, was: alanjay
changed on 10/4/05, was: alan123

That would be incredibly cool - and way better than my duplication records!

Yes, but as you said - it's easy to make work for someone else - I have a feeling this kind of thing would be incredibly complicated to code. ...
Actually, that was one of the things I tried when I was playing with PHP and MySQL a couple of years ago.  I wanted to make a database of software purchasers (for my apps) and wanted to keep the history of changes (like shown above).

I don't remember the details, but essentially I created a master "person" record, and a series of "change" records.  The person record had the initial data and a unique key.  The change records had the same key, a field identifier ("what changed"), a date ("when it changed") and the new data.

So I could do something like "give me all the changes for person 123, sorted by field identifier and sub-sorted by date.   Then display it as above; the initial data, with each change.   (ok, this'd display oldest-to-newest and above it's newest-to-oldest, but you get the idea)

I was pleased with how easy it was - just wish I'd continued working on it after my vacation ended.  (yeah, I was at the beach with the family - and I was messing with my notebook, Apache, MySQL, and PHP - family thinks I'm nuts, but I was having fun!  :)

Over to you, now - I'm heading to bed!

- Al -

Offline tanstaafl

  • God Member
  • ******
  • Posts: 1363
Synch Settings
« Reply #12 on: October 09, 2005, 07:13:05 PM »
Quote
Quote
lol - haven't used that tip yet, so wasn't thinking about it... but in all seriousness, you already *had* those passwords - you just told PM about them - so could you then really blame PM for losing something that it wasn't responsible for creating in the first place could you?
*Absolutely* - if PM is going to be more than a "simple" password generator. If it'll be a place I put all my secret information then it darn well better keep it all safe and be reliable in doing so! Just imagine the horror of needing those "launch the nukes" codes, and your darn password generator lost 'em 'cause "I didn't make it so it's not important if I lose it..."
Well, I think that is the problem... you are trying to use PM for something it was not intended to be. It is not a 'secrets STORAGE' application - it is a password *generator*. It doesn't store *passwords*, it stores *Settings* which simply enables you to reliably *reproduce* your passwords on demand - and as a nifty bonus, even enters them for you, and will soon even auto-sublit them...

Quote
And (I argue) if PM isn't going to store all my secret info, then I need another app to keep them in - and why would I want multiple places for secret stuff? If I've gotta have a Master Secret Repository, then I'll just keep my passwords in that too.
I cannot speak for Eric - if he decides to build some kind of secure 'storage' capability into PM, then I'd have to see how he implemented it before I decided to argue against it - but I do not share your need for PM to be a 'kitchen sink' utility, in spite of the fact that that Feature Request is on the list... ;)

Quote
It goes back to wanting *one* place to keep things - with duplication on my PDA 'cause that's always with me (but it can't fill out the web site when I'm on the notebook, so it needs to be on the notebook too...)
Again, I think you are expecting too much from PM - but thats just me...

Quote
I don't agree on the master server though - I don't think it's necessary, and if there's a boss machine, what if that one's not available for some reason? If it's all peer-to-peer, then one can drop out and it's no big deal.

True Multiple Master Replication functionality is very complex and very difficult to implement correctly. A single Master concept is *much* simpler to implement, and issues of it not being available are easily dealt with using LDAP.

Quote
Would LDAP sync a user's complete information in one blob? Or record-by-record? (PM account-by-account)
Field by field - it is a real Database. I'm not sure how hard or easy it ould be to implement capturing the histiry of all changes to all fields, though - and though it would be 'nice', I don't think it is important enough to even be on the table at this stage.

Quote
I got the impression that LDAP was unable to have information linked to two places - like an account under two groups (that's planned soon, right?) Would LDAP handle that?
Of course - Samba can use LDAP as a back end and it works very well, and User Accounts can be in as many Groups as desired.

Quote
Quote
This would give us *field* level control of modifications, in multiple directions and from multiple sources.

<snip>

 when I was playing with PHP and MySQL a couple of years ago. I wanted to make a database of software purchasers (for my apps) and wanted to keep the history of changes (like shown above).

I don't remember the details, but essentially I created a master "person" record, and a series of "change" records. The person record had the initial data and a unique key. The change records had the same key, a field identifier ("what changed"), a date ("when it changed") and the new data.

So I could do something like "give me all the changes for person 123, sorted by field identifier and sub-sorted by date. Then display it as above; the initial data, with each change. (ok, this'd display oldest-to-newest and above it's newest-to-oldest, but you get the idea)

I was pleased with how easy it was - just wish I'd continued working on it after my vacation ended. (yeah, I was at the beach with the family - and I was messing with my notebook, Apache, MySQL, and PHP - family thinks I'm nuts, but I was having fun!
Sounds very interesting. I still think LDAP is ideally suited to this, but Eric will have to decide. I just wish I could help with the programming...

Offline ajw

  • Jr. Member
  • **
  • Posts: 81
Synch Settings
« Reply #13 on: October 09, 2005, 09:08:02 PM »
Quote
Well, I think that is the problem... you are trying to use PM for something it was not intended to be. It is not a 'secrets STORAGE' application - it is a password *generator*. It doesn't store *passwords*, it stores *Settings* which simply enables you to reliably *reproduce* your passwords on demand - and as a nifty bonus, even enters them for you, and will soon even auto-sublit them...
Yup, that's me - always pushing for more...   :)

But that really does define it - is PM just going to generate passwords or will it evolve to more?  (Eric'll have to decide - and he may not even know right now!  :)

The problem with just generating them is just what Tyr wrote in the FRL:
Anyway, I'll update and crud when I get back. I can't do it now 'cause I can't remember what settings I used for my passwords.

He can't update 'cause he can't generate the right password!  To a forum it's no big deal, but what if it was for his online banking?  And the mortgage check is going to bounce unless money is transferred....

When I'm here at my notebook, PM as it is now is fine.  But how do I bring all that with me?  *Possibly* the on-line version - if you remember all the correct settings for that particular account.   Or, perhaps on a PDA?  
 ;)

By no means do I disparage Eric's work so far - please nobody take it that way!
I just (kinda sorta strongly) feel that there's more to it, and PM could be that more.

(Btw, the reason I'm looking to replace what I'm using right now is that my current solution doesn't fill in the web form.  It syncs between PDA and desktop, and data's accessible and modifyable on both PDA and desktop)


Quote
I cannot speak for Eric - if he decides to build some kind of secure 'storage' capability into PM ...
He's already said he'll be encrypting the "pre-defined password" field (whatever it'll be called) and he said he could encrypt the description field.  So that's a long way towards the general secure storage already.  (I'm not looking for NSA-level security, btw - just that if someone steals my notebook or PDA, they don't get my bank account and credit cards in the bargain)


Quote
True Multiple Master Replication functionality is very complex and very difficult to implement correctly. A single Master concept is *much* simpler to implement, and issues of it not being available are easily dealt with using LDAP.
I'll still argue that I don't think multiple replication would be that difficult given the small amounts of data we're dealing with (as opposed to, say, a credit-card database, or source-code files for a large project).   But I don't think it's so important that it's needed, either.

I'll still have to plead ignorance about LDAP - I just don't know enough to speak about it.  And I'm not much better with databases in general - some playing years and years ago, and a smattering of messing around with MySQL.  I could be totally off-base on what's easy/difficult or even possible.

If LDAP does what's necessary, that's fine by me!

- Al -

Offline tanstaafl

  • God Member
  • ******
  • Posts: 1363
Synch Settings
« Reply #14 on: October 10, 2005, 01:38:00 AM »
Quote
The problem with just generating them is just what Tyr wrote in the FRL:
Anyway, I'll update and crud when I get back. I can't do it now 'cause I can't remember what settings I used for my passwords.

He can't update 'cause he can't generate the right password! To a forum it's no big deal, but what if it was for his online banking? And the mortgage check is going to bounce unless money is transferred....

When I'm here at my notebook, PM as it is now is fine. But how do I bring all that with me? *Possibly* the on-line version - if you remember all the correct settings for that particular account. Or, perhaps on a PDA?
Or you could think ahead a little and bring your RDF file with you on a flash drive, or email it to yourself via an account that you specifically know the password for.

Come on - there is no perfect solution for something like this...

What if your PDA dies?

Quote
By no means do I disparage Eric's work so far - please nobody take it that way! I just (kinda sorta strongly) feel that there's more to it, and PM could be that more.
Maybe, but adding secure 'storage' to PMs list of features will in no way solve Tyr's particular problem either.

Quote
He's already said he'll be encrypting the "pre-defined password" field
Haven't heard of this feature... whats that?

Quote
I'll still argue that I don't think multiple replication would be that difficult given the small amounts of data we're dealing with (as opposed to, say, a credit-card database, or source-code files for a large project). But I don't think it's so important that it's needed, either.
It's not the amount of data that is at issue - it is the functionality itself. You can get this kind of functionality with an Enterprise Oracle Database - but at what cost?

PostgreSQL has an Replication Server, but have no idea the extent of its capabilities.

Anyway, we're a ways from this, so there's plenty of time for debate.

PasswordMaker Forums

Synch Settings
« Reply #14 on: October 10, 2005, 01:38:00 AM »