Replication

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Replication

Eric-2
Picking up a couple of sentences from a different thread, just for a
reference point really, please feel free to ignore ...

On Thu, 11 Oct 2018 10:20:08 -0600, Warren Young <[hidden email]> wrote:
> On Oct 11, 2018, at 12:26 AM, Darren Duncan <[hidden email]> wrote:
8>< --------

>> This makes me think that it would be useful, if it doesn't already,
>> for Fossil to have something analogous to a database replication feature.
>
> That's pretty much what Fossil *is*: a replicated database.  Most of
> it happens to be blockchain structured, rather than relational table
> structured, but much of the table-structured data is also synchronized
> between a clone and its parent.

Database replication that could be applied to any SQLite database might
be a very useful thing, but the Fossil synchronization algorithm is not
much help for that, and that sort of database replication is not a lot
of use for Fossil.

A Fossil repository is a collection of immutable artifacts each named
by a cryptographic hash of what it contains, which is metadata and/or
content, the latter being a file, or a list of the artifacts in a checkin,
or some other things. Artifacts are tied together by the metadata,
which contains the names of other artifacts. There are no deletions,
and no updates in place (well, there is shunning, but that is a special
case for a special purpose). There are quite a few types of artifact.

The artifacts are stored as blobs in a table in a SQLite database. There
are other tables in the database, but almost all of them can be, and are,
populated from the metadata in the artifacts. Fossil synchronization
is essentially an exchange of artifacts, followed by each repository
populating tables from the metadata from the newly-received artifacts. The
"exchange" part of the synchronization knows nothing about tables, rows,
and columns.

The use of a "relational" database to store a Fossil repository is
sort-of incidental.

This is nothing like database replication as generally understood, which
is commonly done by applying redo (write-ahead) logs from the other side,
but it is exactly what Fossil needs.

The interlocking of artifacts by cryptographic hashes does seem very much
like the same idea as blockchain, which Wikipedia says was invented in
2008. It is interesting that the first Fossil checkin was 21 July, 2007
(and the first git checkin was 7 April, 2005).

WARNING: the following sentence will be claimed to be controversial:

No database based on SQL is truly relational.

But if it was, you could treat a row as an artifact named by
table-name+primary-key, ban deletions and updates-in-place, and use
Fossil-style synchronization for database replication. Not realistic
for most existing databases though.

Eric
--
ms fnd in a lbry
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Replication

Keith Medcalf

Balderdash.

> The interlocking of artifacts by cryptographic hashes does seem very much
> like the same idea as blockchain, which Wikipedia says was invented in
> 2008. It is interesting that the first Fossil checkin was 21 July, 2007
> (and the first git checkin was 7 April, 2005).

Hashed Double Linked Lists (blockchain) was invented in the 1950's.  Just many of you are too young to remember your history.

---
The fact that there's a Highway to Hell but only a Stairway to Heaven says a lot about anticipated traffic volume.




_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Replication

Eric-2
On Thu, 11 Oct 2018 14:37:47 -0600, "Keith Medcalf" <[hidden email]> wrote:

>
> Balderdash.
>
> > The interlocking of artifacts by cryptographic hashes does seem very much
> > like the same idea as blockchain, which Wikipedia says was invented in
> > 2008. It is interesting that the first Fossil checkin was 21 July, 2007
> > (and the first git checkin was 7 April, 2005).
>
> Hashed Double Linked Lists (blockchain) was invented in the 1950's.
> Just many of you are too young to remember your history.

I may actually be too old to remember! Do you happen to have a reference
for it? My actual point was that Fossil is now described as blockchain
when it predates what we (currently) call blockchain, and so do other similar
things.

Eric
--
ms fnd in a lbry
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Replication

Warren Young
In reply to this post by Eric-2
On Oct 11, 2018, at 2:25 PM, Eric <[hidden email]> wrote:
>
> On Thu, 11 Oct 2018 10:20:08 -0600, Warren Young <[hidden email]> wrote:
>> On Oct 11, 2018, at 12:26 AM, Darren Duncan <[hidden email]> wrote:
> 8>< --------
>
>>> This makes me think that it would be useful, if it doesn't already,
>>> for Fossil to have something analogous to a database replication feature.
>>
>> That's pretty much what Fossil *is*: a replicated database.

[snip]

> This is nothing like database replication as generally understood, which
> is commonly done by applying redo (write-ahead) logs from the other side,
> but it is exactly what Fossil needs.

I agree that what Fossil does is not the same thing as general-purpose relational database replication, but it doesn’t need to be general-purpose.  Fossil’s synchronization mechanism is custom-tailored to its specific purpose.

If you were hoping to use Fossil as a general-purpose SQLite replication system, then yeah, it’s not going to work for you.  You might want to look at Bedrock:

    http://bedrockdb.com/

> The interlocking of artifacts by cryptographic hashes does seem very much
> like the same idea as blockchain

Relevant: https://fossil-scm.org/index.html/doc/trunk/www/blockchain.md

I prefer the term Merkle tree, as it gets you away from all the hype around cryptocurrencies, but drh prefers blockchain, so that’s what I use now when talking about Fossil.

    https://en.wikipedia.org/wiki/Merkle_tree
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Replication

R Smith-2
In reply to this post by Eric-2

> WARNING: the following sentence will be claimed to be controversial:
>
> No database based on SQL is truly relational.

LOL - who would claim that to be controversial?

It doesn't spur controversy...

It's worthy of a shrug at best, perhaps a "So what?".


It sounds like a deepity - much like any of these:
- Nothing is ever really True...
- Is reality even really real?
- No ice-cream machine ever makes TRUE ice-cream.

An SQL database is deemed "Relational" when it can communicate mildly
relational data using mildly relational (but mathematically sound)
methods. It doesn't need to be (nor claim to be) the Almighty keeper of
all relationality, nor even simply conform to various specific
interpretations of the word "Relation".

In case the point still eludes: We call an SSL hand-shake such because
it behaves by mutual agreement - much like a human hand-shake - but just
because we call it so, doesn't bestow upon it a necessity to behave in
every way like a literal hand-shake, lest some pipe-smoking mountain
wisdom gazes far in the distance while stroking grey beard slowly and
declaring: "no SSL hand-shake is truly hand-shaky".




PS: While I feel some ambivalence towards the subject, I was nodding in
agreement with most of your post, till that line appeared.   :)

PPS: Apologies for inventing some words there....

PPPS: Thumbs up for the Bedrock suggestion from another post - that
system really rocks.



_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Replication

David Barrett-4
Incidentally, Bedrock is built on a blockchain as well -- though I agree
with the sentiment that blockchain isn't actually new at all, and not that
big of a deal.  More information is here:
http://bedrockdb.com/blockchain.html  Hope you enjoy it!

-david

On Thu, Oct 11, 2018 at 3:06 PM R Smith <[hidden email]> wrote:

>
> > WARNING: the following sentence will be claimed to be controversial:
> >
> > No database based on SQL is truly relational.
>
> LOL - who would claim that to be controversial?
>
> It doesn't spur controversy...
>
> It's worthy of a shrug at best, perhaps a "So what?".
>
>
> It sounds like a deepity - much like any of these:
> - Nothing is ever really True...
> - Is reality even really real?
> - No ice-cream machine ever makes TRUE ice-cream.
>
> An SQL database is deemed "Relational" when it can communicate mildly
> relational data using mildly relational (but mathematically sound)
> methods. It doesn't need to be (nor claim to be) the Almighty keeper of
> all relationality, nor even simply conform to various specific
> interpretations of the word "Relation".
>
> In case the point still eludes: We call an SSL hand-shake such because
> it behaves by mutual agreement - much like a human hand-shake - but just
> because we call it so, doesn't bestow upon it a necessity to behave in
> every way like a literal hand-shake, lest some pipe-smoking mountain
> wisdom gazes far in the distance while stroking grey beard slowly and
> declaring: "no SSL hand-shake is truly hand-shaky".
>
>
>
>
> PS: While I feel some ambivalence towards the subject, I was nodding in
> agreement with most of your post, till that line appeared.   :)
>
> PPS: Apologies for inventing some words there....
>
> PPPS: Thumbs up for the Bedrock suggestion from another post - that
> system really rocks.
>
>
>
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Replication

Eric-2
On Thu, 11 Oct 2018 16:56:21 -0700, David Barrett <[hidden email]> wrote:
> Incidentally, Bedrock is built on a blockchain as well -- though I agree
> with the sentiment that blockchain isn't actually new at all, and not that
> big of a deal.  More information is here:
> http://bedrockdb.com/blockchain.html  Hope you enjoy it!

Thankyou, I did, except for their use of the word "query" to mean
inserts, updates, and deletes :-)

Eric
--
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Replication

Eric-2
In reply to this post by R Smith-2
On Fri, 12 Oct 2018 00:06:38 +0200, R Smith <[hidden email]> wrote:

>
>> WARNING: the following sentence will be claimed to be controversial:
>>
>> No database based on SQL is truly relational.
>
> LOL - who would claim that to be controversial?
>
> It doesn't spur controversy...
>
> It's worthy of a shrug at best, perhaps a "So what?".

You may say that, many others would disagree.

8>< --------

> An SQL database is deemed "Relational" when it can communicate mildly
> relational data using mildly relational (but mathematically sound)
> methods. It doesn't need to be (nor claim to be) the Almighty keeper of
> all relationality, nor even simply conform to various specific
> interpretations of the word "Relation".

There is no such thing as relational data, data is what it is and the
Relational Model (E.F.Codd ...) is a way of organising data (any data,
despite many arguments to the contrary). Using "relational" as a sort of
generic term, or as a metaphor, leads to confusion because it hides the
fact that there is a formal system which is behind the creation of SQL
databases, except that the various implementers got it wrong, including
missing out some key points of the Model.

8>< --------

> PS: While I feel some ambivalence towards the subject, I was nodding in
> agreement with most of your post, till that line appeared.   :)

Thankyou!

> PPS: Apologies for inventing some words there....

I have no problem with that.

Eric
--
ms fnd in a lbry
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Replication

Eric-2
In reply to this post by Warren Young
On Thu, 11 Oct 2018 15:44:23 -0600, Warren Young <[hidden email]> wrote:

> On Oct 11, 2018, at 2:25 PM, Eric <[hidden email]> wrote:
>>
>> On Thu, 11 Oct 2018 10:20:08 -0600, Warren Young <[hidden email]> wrote:
>>> On Oct 11, 2018, at 12:26 AM, Darren Duncan <[hidden email]> wrote:
>> 8>< --------
>>
>>>> This makes me think that it would be useful, if it doesn't already,
>>>> for Fossil to have something analogous to a database replication feature.
>>>
>>> That's pretty much what Fossil *is*: a replicated database.
>

[following piece of snip restored]

>> Database replication that could be applied to any SQLite database might
>> be a very useful thing, but the Fossil synchronization algorithm is not
>> much help for that, and that sort of database replication is not a lot
>> of use for Fossil.

> [snip]
>
>> This is nothing like database replication as generally understood, which
>> is commonly done by applying redo (write-ahead) logs from the other side,
>> but it is exactly what Fossil needs.
>
> I agree that what Fossil does is not the same thing as general-purpose
> relational database replication, but it doesn't need to be
> general-purpose.  Fossil's synchronization mechanism is custom-tailored
> to its specific purpose.

This is what I was actually saying.

> If you were hoping to use Fossil as a general-purpose SQLite replication
> system, then yeah, it's not going to work for you.

No, I wasn't, but others were, e.g. thread "How does fossil synchronize
databases?" in the comp.lang.tcl newsgroup.

> You might want to look at Bedrock:
>
>     http://bedrockdb.com/

I will.

>> The interlocking of artifacts by cryptographic hashes does seem very much
>> like the same idea as blockchain
>
> Relevant: https://fossil-scm.org/index.html/doc/trunk/www/blockchain.md
>
> I prefer the term Merkle tree, as it gets you away from all the hype
> around cryptocurrencies, but drh prefers blockchain, so that's what
> I use now when talking about Fossil.
>
>     https://en.wikipedia.org/wiki/Merkle_tree

I agree with the idea of not using the term blockchain, but one of the
reasons for using it is that everyone has heard of it :-( .

Eric
--
ms fnd in a lbry
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Replication

R Smith-2
In reply to this post by Eric-2

>> An SQL database is deemed "Relational" when it can communicate mildly
>> relational data using mildly relational (but mathematically sound)
>> methods. It doesn't need to be (nor claim to be) the Almighty keeper of
>> all relationality, nor even simply conform to various specific
>> interpretations of the word "Relation".
> There is no such thing as relational data, data is what it is and the

You call it "pink sky-monkey data" then, I will call it relational data
when it is structured to contain records that stand in relation to other
records. My contention is not to the name of the data but to its
use/function/storage arrangement. Actually, never-mind, I'll concede the
point. It's just data.

> Relational Model (E.F.Codd ...) is a way of organising data (any data,
> despite many arguments to the contrary). Using "relational" as a sort of
> generic term, or as a metaphor, leads to confusion because it hides the
> fact that there is a formal system which is behind the creation of SQL
> databases, except that the various implementers got it wrong, including
> missing out some key points of the Model.

Again... We know. *shrugs*.

Your basic thesis here revolves around "SQL engines do not follow the
letter and the law of the Relational model" - and the reply, same is
before, is "We know. So what?".
(I'm not disagreeing, I'm just not convinced of controversy)

Let me qualify that: Sure you can kick a person who feels controversial
out from behind any Bush, but if such a controversionist had a real
point, then please ask them to provide a select query example which one
of the current engines cannot solve, but which an engine that followed
the REAL path would be able to solve. I'd venture that if you can find
such a real example with real-world application, then mainline DB
engines would quickly incorporate/adopt it.

I can't speak for everyone, but it is my sincere belief that every DB
engine, at the start, intended to be "The One" that was going to be
closest to the relational model and/or the SQL standard - right before
reality intervened.


Cheers,
Ryan

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Replication

Eric-2
On Fri, 12 Oct 2018 14:31:10 +0200, R Smith <[hidden email]> wrote:
>
> >> An SQL database is deemed "Relational" when it can communicate mildly
> >> relational data using mildly relational (but mathematically sound)
> >> methods. It doesn't need to be (nor claim to be) the Almighty keeper of
> >> all relationality, nor even simply conform to various specific
> >> interpretations of the word "Relation".

>> There is no such thing as relational data, data is what it is and the
8>< --------
> I will call it relational data when it is structured to contain records
> that stand in relation to other records.

Relational databases, and the Relational Model, are not so called because
their records stand in relation to other records. The Model, and the
subsequent databases, are about relations, which are a long-standing
and precisely defined mathematical concept. So, I'm afraid, you are
actually wrong (in common with many others of course).

8>< --------
> its use/function/storage arrangement.

Use and function can be put together, but not storage management with
them. You can have any type of storage management you can think of as
long as it allows the use and function and is efficient enough. This is
one of the points of the Relational Model which almost all SQL-based
databases ignore, they only have one or two storage arrangements, so
"relational" products get blamed for bad performance when it is the design
and implementation of the storage management that causes the problem,
not the relational theory that is claimed to be behind the product.

> Actually, never-mind, I'll concede the point. It's just data.

*shrug*

8>< --------
> Your basic thesis here revolves around "SQL engines do not follow the
> letter and the law of the Relational model" - and the reply, same is
> before, is "We know. So what?".
> (I'm not disagreeing, I'm just not convinced of controversy)

"So what" is that they and their users do not reap the benefits of the
model. Also they may produce incorrect results and we can not prove that
they don't. Any data retrieval that was not anticipated when the system
was designed may perform very badly or even be impossible.

> Let me qualify that: Sure you can kick a person who feels controversial
> out from behind any Bush, but if such a controversionist had a real
> point, then please ask them to provide a select query example which one
> of the current engines cannot solve, but which an engine that followed
> the REAL path would be able to solve. I'd venture that if you can find
> such a real example with real-world application, then mainline DB
> engines would quickly incorporate/adopt it.

Controversy requires only that there are outspoken people on both sides
of an argument, not that either side has any real points. You seem to
have assumed that any controversionists (your invented word) were on my
side, whereas I intended to discourage the more reasonable of those on
the other side from wading in.

> I can't speak for everyone, but it is my sincere belief that every DB
> engine, at the start, intended to be "The One"

Do you remember that there was once a short-lived product actually
called "The Last One"?

> that was going to be closest to the relational model and/or the SQL
> standard

Just "or", the standard itself contradicts the model.

> - right before reality intervened.

Reality can be cruel and illogical and take no regard for the best
interest of anyone involved.

At which point we should stop being quite so OT before someone stamps on
us ;-)

Eric
--
ms fnd in a lbry
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Replication

R Smith-2

> Relational databases, and the Relational Model, are not so called because
> their records stand in relation to other records. The Model, and the
> subsequent databases, are about relations, which are a long-standing
> and precisely defined mathematical concept. So, I'm afraid, you are
> actually wrong (in common with many others of course).

I was defending my use of the the phrase "Relational data" (and even
conceded it), not arguing about the meaning of Relational model - and we
call Relational databases so because it does Relational things with
data, things described in the Relational model... but not necessarily
all the 12 things Edgar wanted us/them to conform to before allowing the
term "Relational" - and that's ok... Surely.

Every word I speak in this exchange seems to serve only as food to feed
a next construed difference, so I'll concede everything and return to
the original observation:

Perhaps it's a cultural difference or linguistic weirdness, but in my
tongue "controversial"  means having real controversy, two sides to a
coin, often hotly contested. i.e. because there are some people who
believe the Earth to be flat, doesn't make the theory of Gravity a
debated controversy.

If we call anything that have a couple of naysayers a controversy, then
everything is a controversy.


Cheers,
Ryan

PS: You're right about one important thing, this thread must be boring
everyone to bits, so I'll rest the case here. :)

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Replication

Keith Medcalf
In reply to this post by R Smith-2

>>> An SQL database is deemed "Relational" when it can communicate
>>> mildly
...

SQL stands for Structured Query Language.

It has nothing whatsoever to do with the data store but rather is a specification of the Language used to retrieve/manipulate the datastore.

This is the same as "C" or "FORTRAN" being language specifications for how a problem is stated to a computer and DOES NOT say anything nor impose any requirements on the underlying CPU architecture (RISC?  CISC?  DEC?  IBM?  Intel?  etc) nor how the program is stored and run (RAM?  Paper Tape?  Core?  Drum?).

You can have "SQL" used to query a "relational database", query a hierarchical database, query a network or network-extended model database.  You can use "navigational calls" to navigate a "relational database", "hierarchical database", or a "network or network-extended database".

>> Relational Model (E.F.Codd ...) is a way of organising data (any
>> data, despite many arguments to the contrary). Using
>> "relational" as a sort of generic term, or as a metaphor,
>> leads to confusion because it hides the fact that there is a
>> formal system which is behind the creation of SQL databases,
>> except that the various implementers got it wrong, including
>> missing out some key points of the Model.

Actually, this is incorrect.  a "relational database" is one in which the "relations" are expressed by the storage of "duplicate data" in the various record structures, and that relations are based on the value of the data itself and nothing else.  That is to say that a Bill-of-Material does not contain a "master record" with a "linked list" of "child records" pointing to their "owners" (as in a set), but rather that the "master record" is linked to its children by the duplication of data.  Relational Algebra is the mathematic theory by which this hunk of duplicated data can be designed and manipulated in a cohesive manner without introducing "anomalies" into the processing, such consistency having previously been imposed directly by designing the database using hierarchical, network, or network-extended data models (which are more accurate portrayals (models) of the structure of the real world) ... however, the use of those methods had "issues" of their own that could be solved by using a "relational data duplication" model (in that the relationships are inherent in the data values themselves and not dependent on some artificial data structure which cannot be easily re-created if corrupted) and reflected the increase in storage and processing power that was necessary to maintain all that duplication.

>Your basic thesis here revolves around "SQL engines do not follow the
>letter and the law of the Relational model" - and the reply, same is
>before, is "We know. So what?".
>(I'm not disagreeing, I'm just not convinced of controversy)

That is because SQL (Structured Query Language) is an imperative Query Language.  It does not describe the underlying datastore, which may or may not be based on Relational Algebra, but is merely a Query Language -- a way to express what it is that one wants to be done.

>I can't speak for everyone, but it is my sincere belief that every DB
>engine, at the start, intended to be "The One" that was going to be
>closest to the relational model and/or the SQL standard - right
>before reality intervened.

No, the Database Engine (as in the data storage layer) intends to provide a way of modelling the "real world".  The relational model is simply a method by which mathematical concepts can be applied to prove correctness of the model.  That is to say that a "relational database" means that the datastore is "relational" and can be manipulated in accordance with the postulates of relational algebra and is, under those postulates, correct and anomaly free.

SQL is merely the manner in which one manipulates the datastore, it does not matter the actual format of storage of the datastore since, in the end, correctness of the data model can only be proved by mathematical means.




_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users