Newbie coherency questions ... poor man's replication? Proper replication approaches?

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Newbie coherency questions ... poor man's replication? Proper replication approaches?

Bryan J. Smith
I've heard about this project for so many years, but I don't
know why I ignored it until now.  I was just reading the FAQ
on coherency and threading:  
  http://www.sqlite.org/faq.html#q7 

I believe this is outstanding for an embedded DB engine,
something I've never seen in anything else.  If I am assuming
correctly, while SQLite may not be a "true" multi-write DB in
the eyes of some -- I assume by having difference processes
locking control access to the DB, you can effectively have
multi-write (just one at a time), correct?

I assume that only works between programs using SQLite
methods in their code?  Or does it mean I could possibly test
for control, lock the DB and even rsync the DB file remotely?
 I assume not, but I have to ask if this "poor man's
replication" approach could work.  Assuming that would not,
so read on (and please comment on) what I believe might be
more proper design.

We're building a multi-master, peer-to-peer network service,
so we need a way to replicate changes between systems.
Before finding SQLite, I figured I'd have to write an
"all-in-one daemon."  E.g., an application that handles the
app's data itself with client service access plus peer
replication service access to the same DB -- probably with
threads, semaphores and other internal locking issues.

But now I think SQLite solves the problem nicely, if I'm
reading the FAQ correctly.  What I'm considering is a 2
daemon approach, one for the app/client services and another
for the replication services.  I would just thread as many
daemons as configured for each (creating a new SQLite DB
connection instance to the file for each thread, per FAQ #8),
and handle client service changes and replication service
changes appropriately with a write lock.

If that is not feasible/recommended, alternatively, I was
thinking of keeping separate databases for the peer
replication, whereby the app/client services daemon and
replication services daemon never have write access to each
other's databases.  There would be incoming databases of
changes from each peer, and a single outgoing database for
all other peers (of local changes)

Or should I even be looking at maintaining local copies of
remote databases, possibly using an rsync at startup, and
then doing local resolution?  I guess there's a lot of
options/issues here, and I'm looking for suggestions/best
practices.

I know this is a broad set of questions/commentary, but any
insight into what SQLite can and can't do, and what practices
are best for setting up a multi-master, peer-replicating
service (of which, each "master" is also servicing clients,
which are where the changes initially come from) would be
ideal.

Thanx in advance.


--
Bryan J. Smith                | Sent from Yahoo Mail
mailto:[hidden email]     |  (please excuse any
http://thebs413.blogspot.com/ |   missing headers)
Reply | Threaded
Open this post in threaded view
|

Re: Newbie coherency questions ... poor man's replication? Proper replication approaches?

Dan Kennedy

> We're building a multi-master, peer-to-peer network service,
> so we need a way to replicate changes between systems.

Unsolicited advice:

Of course I don't understand your problem domain, but I'm a
big fan of a technology called "virtual synchrony" for this
kind of thing. Very successful when I used it in a closed
source project, and I believe the same technology used by
the postgresql database for replication. (Google for Ken
Birman, a researcher into the same).

http://www.spread.org is a very good open-source implementation
of his model. Another system is called "Ensemble", although I
found Spread to be better documented.

As I say, completely unsolicited.



               
__________________________________
Yahoo! Mail - PC Magazine Editors' Choice 2005
http://mail.yahoo.com
Reply | Threaded
Open this post in threaded view
|

Re: Newbie coherency questions ... poor man's replication? Proper replication approaches?

Jay Sprenkle
In reply to this post by Bryan J. Smith
On 10/3/05, Bryan J. Smith <[hidden email]> wrote:

>
>
>
> We're building a multi-master, peer-to-peer network service,
> so we need a way to replicate changes between systems.
> Before finding SQLite, I figured I'd have to write an
> "all-in-one daemon." E.g., an application that handles the
> app's data itself with client service access plus peer
> replication service access to the same DB -- probably with
> threads, semaphores and other internal locking issues.
>

You're still going to have to understand locking semaphores and
threads. All sqlite does for you is give you a good interface to access
the data, transactions, and hardware fault tolerance.
Reply | Threaded
Open this post in threaded view
|

Re: Newbie coherency questions ... poor man's replication? Proper replication approaches?

Bryan J. Smith
Jay Sprenkle <[hidden email]> wrote:
> You're still going to have to understand locking semaphores
> and threads.

Yeah, I was thinking about that.
There are still structures in memory and other access issues.

> All sqlite does for you is give you a good interface
> to access the data, transactions, and hardware fault
> tolerance.

Yep.  I forgot some of my fundamentals there for a second.


--
Bryan J. Smith                | Sent from Yahoo Mail
mailto:[hidden email]     |  (please excuse any
http://thebs413.blogspot.com/ |   missing headers)