dqlite 1.0.0 - SQLite replication and failover library

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

dqlite 1.0.0 - SQLite replication and failover library

Free Ekanayaka
Hi,

following up from my previous post back in 2017 [0], I'd like to
announce version 1.0.0 of dqlite, a C library that brings data
replication and high-availability to SQLite, using the Raft consensus
algorithm.

The biggest change is that Go is not used anymore, the engine itself is
all pure C now.

It still requires to apply a patch to SQLite, but it's a rather tiny one
that just adds a few hooks when writing to the Write-Ahead log.

See https://dqlite.io for more details.

Thanks again to the SQLite authors for their excellent work.

Cheers,

Free

[0] http://mailinglists.sqlite.org/cgi-bin/mailman/private/sqlite-users/2017-August/074384.html
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: dqlite 1.0.0 - SQLite replication and failover library

test user
Hey Free,

Looks like an interesting project.

Is there a blog or docs about the reasons for the move from Go to C?

Also what types of systems would utilise dqlite? Are there current users?

Thanks


On Thu, 29 Aug 2019 at 11:41, Free Ekanayaka <[hidden email]> wrote:

> Hi,
>
> following up from my previous post back in 2017 [0], I'd like to
> announce version 1.0.0 of dqlite, a C library that brings data
> replication and high-availability to SQLite, using the Raft consensus
> algorithm.
>
> The biggest change is that Go is not used anymore, the engine itself is
> all pure C now.
>
> It still requires to apply a patch to SQLite, but it's a rather tiny one
> that just adds a few hooks when writing to the Write-Ahead log.
>
> See https://dqlite.io for more details.
>
> Thanks again to the SQLite authors for their excellent work.
>
> Cheers,
>
> Free
>
> [0]
> http://mailinglists.sqlite.org/cgi-bin/mailman/private/sqlite-users/2017-August/074384.html
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: dqlite 1.0.0 - SQLite replication and failover library

Free Ekanayaka
Hello,

I updated the FAQ [0] with your two questions.

Free

[0] https://github.com/canonical/dqlite/blob/master/doc/faq.md#why-c

test user <[hidden email]> writes:

> Hey Free,
>
> Looks like an interesting project.
>
> Is there a blog or docs about the reasons for the move from Go to C?
>
> Also what types of systems would utilise dqlite? Are there current users?
>
> Thanks
>
>
> On Thu, 29 Aug 2019 at 11:41, Free Ekanayaka <[hidden email]> wrote:
>
>> Hi,
>>
>> following up from my previous post back in 2017 [0], I'd like to
>> announce version 1.0.0 of dqlite, a C library that brings data
>> replication and high-availability to SQLite, using the Raft consensus
>> algorithm.
>>
>> The biggest change is that Go is not used anymore, the engine itself is
>> all pure C now.
>>
>> It still requires to apply a patch to SQLite, but it's a rather tiny one
>> that just adds a few hooks when writing to the Write-Ahead log.
>>
>> See https://dqlite.io for more details.
>>
>> Thanks again to the SQLite authors for their excellent work.
>>
>> Cheers,
>>
>> Free
>>
>> [0]
>> http://mailinglists.sqlite.org/cgi-bin/mailman/private/sqlite-users/2017-August/074384.html
>> _______________________________________________
>> sqlite-users mailing list
>> [hidden email]
>> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>>
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: dqlite 1.0.0 - SQLite replication and failover library

Jose Isaias Cabrera-4
In reply to this post by Free Ekanayaka

Free Ekanayaka, on Thursday, August 29, 2019 06:40 AM, wrote...

>
> Hi,
>
> following up from my previous post back in 2017 [0], I'd like to
> announce version 1.0.0 of dqlite, a C library that brings data
> replication and high-availability to SQLite, using the Raft consensus
> algorithm.
>
> The biggest change is that Go is not used anymore, the engine itself is
> all pure C now.
>
> It still requires to apply a patch to SQLite, but it's a rather tiny one
> that just adds a few hooks when writing to the Write-Ahead log.
>
> See https://dqlite.io for more details.
>
> Thanks again to the SQLite authors for their excellent work.

Can dsqlite be installed on Windows?  I went to the site, read the README.md file, and could not find any reference of it.  I can see the

$ sudo add-apt-repository -y ppa:dqlite/v1 && sudo apt install dqlite

which lets me know that it linux/unix based.  But, is Windows an option also?  Thanks.

josé

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: dqlite 1.0.0 - SQLite replication and failover library

Dominique Devienne
On Thu, Aug 29, 2019 at 2:35 PM Jose Isaias Cabrera <[hidden email]>
wrote:

> Free Ekanayaka, on Thursday, August 29, 2019 06:40 AM, wrote...
> > See https://dqlite.io for more details.
>
> Can dsqlite be installed on Windows?  I went to the site, read the
> README.md file, and could not find any reference of it.  I can see the
>
> $ sudo add-apt-repository -y ppa:dqlite/v1 && sudo apt install dqlite
>
> which lets me know that it linux/unix based.  But, is Windows an option
> also?  Thanks.
>

Ditto. Wasn't clear is Windows was supported.
But I'd add whether dqlite could replicate across platforms as well.
SQLite DB files are portable across platforms and architectures.
So can dqlite replicate across Windows, Linux, OSX, etc???

Also, does it come in easy-to-digest amalgamation?
Especially since it requires a patch to SQLite, a portable
C-amalgamation source distribution form would maximize
changes of it being tried out.

Perhaps OT, but would the hooks added to the WAL-journaling
ever be up-streamed by Richard, under a build flag for example?
Without it, that's basically "forking" SQLite, which is not ideal... --DD
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: dqlite 1.0.0 - SQLite replication and failover library

Free Ekanayaka
In reply to this post by Jose Isaias Cabrera-4
Hello Jose,

Jose Isaias Cabrera <[hidden email]> writes:

> Free Ekanayaka, on Thursday, August 29, 2019 06:40 AM, wrote...
>>
>> Hi,
>>
>> following up from my previous post back in 2017 [0], I'd like to
>> announce version 1.0.0 of dqlite, a C library that brings data
>> replication and high-availability to SQLite, using the Raft consensus
>> algorithm.
>>
>> The biggest change is that Go is not used anymore, the engine itself is
>> all pure C now.
>>
>> It still requires to apply a patch to SQLite, but it's a rather tiny one
>> that just adds a few hooks when writing to the Write-Ahead log.
>>
>> See https://dqlite.io for more details.
>>
>> Thanks again to the SQLite authors for their excellent work.
>
> Can dsqlite be installed on Windows?  I went to the site, read the README.md file, and could not find any reference of it.  I can see the
>
> $ sudo add-apt-repository -y ppa:dqlite/v1 && sudo apt install dqlite
>
> which lets me know that it linux/unix based.  But, is Windows an
> option also?  Thanks.

At the moment Windows is not an option, mainly because under the hood
dqlite uses a Linux-specifc asynchronous file system API (io_submit) not
available on Windows. However, such code lives behind a pluggable
interface that could be implemented on other OSs.

Free
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: dqlite 1.0.0 - SQLite replication and failover library

Free Ekanayaka
In reply to this post by Dominique Devienne
Hi Dominique,

Dominique Devienne <[hidden email]> writes:

> On Thu, Aug 29, 2019 at 2:35 PM Jose Isaias Cabrera <[hidden email]>
> wrote:
>
>> Free Ekanayaka, on Thursday, August 29, 2019 06:40 AM, wrote...
>> > See https://dqlite.io for more details.
>>
>> Can dsqlite be installed on Windows?  I went to the site, read the
>> README.md file, and could not find any reference of it.  I can see the
>>
>> $ sudo add-apt-repository -y ppa:dqlite/v1 && sudo apt install dqlite
>>
>> which lets me know that it linux/unix based.  But, is Windows an option
>> also?  Thanks.
>>
>
> Ditto. Wasn't clear is Windows was supported.

Added this as well to the FAQ:

https://github.com/canonical/dqlite/blob/master/doc/faq.md#are-windows-and-osx-supported

> But I'd add whether dqlite could replicate across platforms as well.
> SQLite DB files are portable across platforms and architectures.
> So can dqlite replicate across Windows, Linux, OSX, etc???

As said, yes it's possible, but not done. I don't think it would be too
much work.
>
> Also, does it come in easy-to-digest amalgamation?
> Especially since it requires a patch to SQLite, a portable
> C-amalgamation source distribution form would maximize
> changes of it being tried out.

Good point, I have thought about it but didn't not have time to
implement it. I'll definitely provide that option in the near future.

> Perhaps OT, but would the hooks added to the WAL-journaling
> ever be up-streamed by Richard, under a build flag for example?
> Without it, that's basically "forking" SQLite, which is not
> ideal... --DD

Absolutely. I'd love to upstream it at some point. However I'll wait
until I'm 100% sure of the API, the code and the logic, since Richard
and the team are known to be rightfully very conservative when it comes
to contributions.

One bit that I'd like to explore is to have a bit more replication
options than just WAL pages: for instance you could have some kind of
mixed logical/binary replication that streams the operations peformed on
the btree (e.g. I updated this cell with this value), or statement-based
replication when it makes sense. Ideally a mix of all those approaches
with some euristics to automatically decide which one is the best for a
particular transaction.

Free

> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: dqlite 1.0.0 - SQLite replication and failover library

Jose Isaias Cabrera-4
In reply to this post by Free Ekanayaka

> From: Free Ekanayaka, on Thursday, August 29, 2019 10:21 AM, wrote...
>
> Hello Jose,
>
> Jose Isaias Cabrera, on

> > which lets me know that it linux/unix based.  But, is Windows an
> > option also?  Thanks.
>
> At the moment Windows is not an option, mainly because under the hood
> dqlite uses a Linux-specifc asynchronous file system API (io_submit) not
> available on Windows. However, such code lives behind a pluggable
> interface that could be implemented on other OSs.

How about cygwin? :-)  Thanks.

josé


_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users