I'm sorry this is so long - I'm in verbal-spew mode these days...
I think I'm gonna find out if there's a message-lenght limit - and I'll capture the award for the longest posted message...
Hi Tan! (if I may be so informal :)
I love your name! I thought about using as a company name oh so many years ago - but then figured I'd have to explain it to potential clients and somehow it seemed like it could interfere with getting their business... :)
Um, actually, the first three entries (uh, accounts) in my setup *do* have passwords; they're for credit cards and I'd already set up passwords for them and didn't want to go through the bother of changing the passwords on-line, and in FireFox, and in my PC/Palm password app... So losing those *would* be bad for me.
(I'm hoping that PasswordMaker can replace the PC/Palm app I'm using now; it works fine, but it doesn't fill in web forms so I'm looking to replace it. I know PM doesn't have a Palm app yet, but I'll wait a while for it. I'd take a stab at it myself (I write Palm apps anyway) but I don't know much about Java and even less about Java on Palm devices - my background is embedded code and device drivers; gimme low-level C and assembly language and let me code to the metal! :)
Same thing - it'll take me time to digest your response too, but at first peek...
Simple backup/restore is easy - just make a big blob of the data... ("easy" is relative of course - as a friend says "Just a simple matter of programming..." :)
Ok, you've got that - Export/Import (although I've only tried Export; I'll take it as a given I can Import correctly too...)
So now you need a "copy this to the FTP server" - with appropriate security for transmission (HTTPS? SSL? I dunno - I don't know this stuff beyond alphabet soup)
and storage.
Before you go off into the SQL database (not a bad idea, I think - I played with MySQL and had a ton of fun learning, although I never followed up on my own project with it - wouldn't mind having a reason to do so again though, but you don't want a newbie designing the schema - or at least I *hope* you don't! :)
I think the synchronization should support machine-to-machine syncing - that's what PDAs do. They won't sync to the FTP server.
What you are talking about is full 'synchronization'. The first stages of implementation will not involve synchronizing the changes between the files, it will be simple Backup/Restore functionality...
Ok, but you've got that locally with RDF files. Why not at least do the remote backup (FTP site) record-by-record - that leads into full syncing. Otherwise you're pretty much going to throw away the backup-to-FTP code. (except that users could set up their own FTP site, so it wouldn't be a bad feature to have anyway)
Throw in the fields you think you'd need for full-sync - modification date, modification status (modified/deleted/created/unchanged), duplication id (my -1, -2...) and so forth - fill 'em in as expected for a new record and then you'll have 'em for existing customer data when full-sync goes live. If it turns out those fields are totally dumb-as-bricks then you're no worse off than if they weren't there in the first place, so it's only a win, except for the additional disk space.
Unless FTP backup to user's own FTP sites is planned, I think the additional work of backup record-by-record instead of one big file isn't that much additional work - of course I'm not coding it... :)
...SQL Database...
I think this is a great idea - why reinvent the wheel?
2. Define a schema for the PasswordMaker Settings (lot of work there)... be sure to include a 'modification status' field (date/time stamp?) for every data field...
I'm not sure a date/time stamp is ideal.
What if I want to sync with my ISP's PasswordMaker Backup Server - and also with yours (and maybe 8 or ten others just cause I'm making the example difficult...)
What if a clock is off? Or more likely, modifications occur on different machines, and are sync'd to only some of the backup servers. Now you've got a mess, 'cause my ISP's server says the data for my Yahoo backup changed (by my notebook) at 9am, but later at work I changed my desktop and only sync'd with your server so your server says the record changed at 10:30. But the data isn't the same - you need to keep both (what I called duplication number - maybe not the best name for it; maybe "modification number" or something...)
I
think the description I gave with 1345-x (and having -1, -2, -3...) duplicatation records would work in this case. If my mental meandering is right, even if a record (or its duplicates) are changed on multiple clients (notebook, PDA, work, home, etc) are synced with only *some* servers (so servers themselves are out-of-sync with each other) and it's made as messy as possible - think about the -1, -2, -3... records being created as I described, by client changes in several places and then syncing, but add in that some clients are only syncing with some servers (maybe I can't get to my ISP's server from work, for example) so we end up with even the -1, -2, -3 records being modified multiple times. Deleted on some servers by some clients even as other clients have modified them.
I think it'll still end up right once all the syncing stabilizes - effectively after one client syncs with all the servers (and resyncs with servers as necessary, so all the servers are again in sync) and then all the clients sync with any of the servers.
We'll end up with additional duplication records (-5, -6, -7, etc) as the duplication records are modified - and as the clients create "collisions" of duplication records; two clients both modify -2, for example, and then sync to different servers - we now have -2 records on different servers with different data.
But when a client syncs with the first server (with the modified -2), it'll see that -2 is different than its own, so will change its -2 to -5 and download the server's -2 (and uploads its former -2; now -5).
Then it syncs with the other server (with the second modified -2) and it'll again see that its (the client's) record is different, so it changes its local -2 to -5 (it hasn't seen the server's -5 yet) and downloads the server's -2.
When the client sees that the server has a -5, and it's different from its own, it changes its (client's) -5 to -6, downloads the server's -5 and uploads -6. (unless the server has a -6, in which case this happens again; client -6 becomes -7...)
If I'm right, multiple clients changing the same records and syncing with multiple servers will all end up with the same data.
If this is done with timestamps, I think either you'll end up erroneouosly thinking records are newer or older, or you'll think records are different when they're the same, and create spurious duplicates. (what if I change a record at home from X to Y and FTP sync, then change the same record at work from X to Y and FTP sync - will I end up with one or two records? Should be one record, I think; the data's the same, and that's what counts.
I may be wrong - timestamps might work just as well as my duplication number, but I've got a nagging feeling about it...
3. Make the local RDF file simply be a local storage container (XML file of course) for your Settings.
I'm not qualified to comment on this - but that's never stopped me!
Uh, whatever works... :)
4. Write the Synch code, which will in effect perform basically the same, conceptually if not actually, as the Master/Slave functionality in OpenLDAP.
Again, not qualified, so I'll take your word on it.
5. Build a web based interface that could then handle conflicts in some meaningfully intuitive way.
That'd be nice, but is it necessary? I think it'd be better to have conflict resolution handled in the client. Ideally, the user would see the their original record and duplicates side-by-side and be able to merge as desired, but at first even a manual merge jumping back-and-forth between the records would be ok.
No, scratch that - that'd suck. But it'd be ok during server design...
(Oh, it's *so* easy to make work for other people! :)
6. Fail-safe - provide a means for keeping a local backup copy, protected from the Synch Sessions, so that if somehow the data in the DB got hosed, you could just reupload your Settings from your last Backup - and make damn sure that a *validated* fresh Backup is made prior to any Synch Sessions.
Yes! Absolutely! Might even keep a duplicate of the backup on the server - belt-and-suspenders...
Would be cool to be able to roll back the sync - both locally and on the server, but I think that's a nest of worms that make this whole discussion look like a cakewalk. (what if another client syncs before the first client requests a roll-back...)
But a local pre-sync backup is a great idea. Perhaps even keep several aged backups. (lest a user sync with one server, then another, and another - and *then* discover he's hosed)
This would give us *field* level control of modifications, in multiple directions and from multiple sources.
If I read you right, that would allow multiple modifications to be shown in the single record, with multiple data. So instead of two "credit card foo" records (-1, -2 in my nomenclature) I'd see a single "credit card foo" record, and something like:
Name: Credit card foo Visa
changed on 10/4/05, was: foo card
Username: alan
changed on 10/6/05, was: alanjay
changed on 10/4/05, was: alan123
That would be incredibly cool - and way better than my duplication records!
Display of change history would be optional, of course.
Should the user be able to remove the change history? Probably, but I wouldn't care if they couldn't - it's a valuable record, so I don't mind if it's permanant.
(would be on the server, anyway - or should they be able to delete the history there too?)
Record deletions might be tricky - delete on A, sync, modify on B, sync... You'd have to see something like:
Record deleted on 10/1/05 (by client X)
Record restored due to modifications on 10/2/05 (by client Y)
Maybe that history should include all the modifications (optional?) so it'd show:
Record deleted on 10/1/05 (by client X)
Record restored due to modifications on 10/2/05 (by client Y)
Name changed on 10/4/05: was: foo card
changed to: Credit card foo Visa
Username changed on 10/4/05: was: alan123
changed to: alanjay
Username changed on 10/6/05: was: alanjay
changed to: alan123
Uh, I showed the changes in the single-record display above in descending date order, here I showed in ascending date order - probably should be user-configurable... (I'm big on user options - especially when someone else is coding... :)
I'm not saying this would be *easy* of course - definitely a lot harder than simple FTP backup/restore capability - but it might be the best way to code a *reliable* method for accomplishing what we want that will work across all of the different implementations (PDA's coming immediately to mind as behaving differently than the Browser extensions)..
Would PDAs be syncing to the FTP server? I was figuring they'd sync locally, like Palm or PocketPC does now...
The only thing I don't like about it is it forces people to use the PasswordMaker Server if they want to be able to synchronize - at least at first. I guess we would make the DB schemas open as well, so if someone was capable of setting up their own DB server, they could host it themselves - and maybe it would become a common offering for Hosting Companies, along with Mailman and Apache.
Could create a standalone module that contains a web server and DB server (SQLite?) - that would simplify things for end-users who wanted to run their own server. Roundup bug-tracking system does that; I think Trac does too. So it's doable; not sure if the work is worth it though, it *is* easy enough to set up Apache and MySQL. Well,... a single standalone app *would* be easier...
Comments?
Uh, I think I provided a plethora - aren't you sorry you asked?
- Al -