Status of LSM1 extension

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Status of LSM1 extension

Dominique Devienne
Hi,

I'm interested in LSM1 [1] as an alternative to SQLite, since in a
particular use-case,
I'm using SQLite mostly as a key-value store, and write performance is
particularly important,
in addition to MVCC. Sounds like it could be an excellent fit here,
and the fact it comes from
the SQLite team is something I value.

That said, the only online doc for LSM1 ([2] and [3]) are from the
defunct SQLite4 web-site,
and the main blog post is starting to look dated [4]. I thus wonder
about the level of quality
and support on LSM1, and lack of doc for it in the main SQLite web-site.

In terms of practicality, there's also no amalgamation for LSM1. And the virtual
table over LSM1 data-files [5], something I was look for, does not
appear to be documented
anywhere. Notably whether using that vtable using the familiar SQLite
API is advisable
instead of using the different and unfamiliar LSM1-specific API.

I'm just looking for clarity and advice around LSM1, as well as
commitments regarding
its level of quality, testing, and support. And whether we can hope to
have more doc and
amalgamation deliverables in the future. It sounds like it's a really
nice piece of code, but
the fact there's very little noise and advertisement about it is
somewhat worrying.

Thanks, --DD

[1] https://www2.sqlite.org/src/dir?name=ext/lsm1
[2] https://sqlite.org/src4/doc/trunk/www/lsmusr.wiki
[3] https://sqlite.org/src4/doc/trunk/www/lsmapi.wiki
[4] https://charlesleifer.com/blog/lsm-key-value-storage-in-sqlite3/
[5] https://www2.sqlite.org/src/finfo?name=ext/lsm1/lsm_vtab.c
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Status of LSM1 extension

Dan Kennedy-4

On 5/3/63 16:11, Dominique Devienne wrote:

> Hi,
>
> I'm interested in LSM1 [1] as an alternative to SQLite, since in a
> particular use-case,
> I'm using SQLite mostly as a key-value store, and write performance is
> particularly important,
> in addition to MVCC. Sounds like it could be an excellent fit here,
> and the fact it comes from
> the SQLite team is something I value.
>
> That said, the only online doc for LSM1 ([2] and [3]) are from the
> defunct SQLite4 web-site,
> and the main blog post is starting to look dated [4]. I thus wonder
> about the level of quality
> and support on LSM1, and lack of doc for it in the main SQLite web-site.
>
> In terms of practicality, there's also no amalgamation for LSM1. And the virtual
> table over LSM1 data-files [5], something I was look for, does not
> appear to be documented
> anywhere. Notably whether using that vtable using the familiar SQLite
> API is advisable
> instead of using the different and unfamiliar LSM1-specific API.
>
> I'm just looking for clarity and advice around LSM1, as well as
> commitments regarding
> its level of quality, testing, and support. And whether we can hope to
> have more doc and
> amalgamation deliverables in the future. It sounds like it's a really
> nice piece of code, but
> the fact there's very little noise and advertisement about it is
> somewhat worrying.


It's very much a solution looking for a problem at this point. It's not,
as far as I know, used in anything that is too widely deployed.

That said, I don't think it's too bad of an implementation. The
automated tests are reasonably good - although of course not as good as
SQLite's though. And the docs are stored in kind of a ridiculous place
at the moment, but I think they're quite complete.

Not planning to develop this any further unless a big user emerges,
which is not impossible. I do intend to fix any reported bugs though.

Dan.



_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Status of LSM1 extension

Dominique Devienne
On Thu, Mar 5, 2020 at 12:35 PM Dan Kennedy <[hidden email]> wrote:

> On 5/3/63 16:11, Dominique Devienne wrote:
> > I'm interested in LSM1 [1] as an alternative to SQLite [...]
>
> [...], I don't think it's too bad of an implementation. The
> automated tests are reasonably good - although of course not as good as
> SQLite's though. And the docs are stored in kind of a ridiculous place
> at the moment, but I think they're quite complete.
>
> Not planning to develop this any further unless a big user emerges,
> which is not impossible. I do intend to fix any reported bugs though.

Thanks Dan. I appreciate your candor, and support commitment.
Given the above, I'll start with SQLite, and when/if I'm done with what
I need to do, will try to give LSM1 a try, to compare performance, and
report back.

Thanks again, --DD

PS: I'd still very much appreciate an LSM1 amalgamation
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Status of LSM1 extension

Keith Medcalf

On Thursday, 5 March, 2020 05:51, Dominique Devienne <[hidden email]> wrote:

>PS: I'd still very much appreciate an LSM1 amalgamation

cd ext/lsm1
tclsh tool/mklsm1c.tcl

which will write an lsm1.c amalgamation in the current directory (ext/lsm1)

You can append this to the amalgamation and use an EXTRA_INIT hook to initialize it, just like building in any other extension (though you need to define SQLITE_ENABLE_LSM1 in order for the extension code to be compiled)

--
The fact that there's a Highway to Hell but only a Stairway to Heaven says a lot about anticipated traffic volume.




_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Status of LSM1 extension

Charles Leifer
Keith, if you could share a bit more details on how you do that, I'd be
interested.

On Thu, Mar 5, 2020 at 8:43 AM Keith Medcalf <[hidden email]> wrote:

>
> On Thursday, 5 March, 2020 05:51, Dominique Devienne <[hidden email]>
> wrote:
>
> >PS: I'd still very much appreciate an LSM1 amalgamation
>
> cd ext/lsm1
> tclsh tool/mklsm1c.tcl
>
> which will write an lsm1.c amalgamation in the current directory (ext/lsm1)
>
> You can append this to the amalgamation and use an EXTRA_INIT hook to
> initialize it, just like building in any other extension (though you need
> to define SQLITE_ENABLE_LSM1 in order for the extension code to be compiled)
>
> --
> The fact that there's a Highway to Hell but only a Stairway to Heaven says
> a lot about anticipated traffic volume.
>
>
>
>
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Status of LSM1 extension

Keith Medcalf

On Thursday, 5 March, 2020 20:39, Charles Leifer <[hidden email]> wrote:

>Keith, if you could share a bit more details on how you do that, I'd be
>interested.

I presume you mean how to create a "built-in" extension, which is available for all connections, just the built-in functions and modules.

There is a built-in define SQLITE_EXTRA_INIT used in main.c which allows for a "user function" to be called at the end of the SQLite3 initialization process.  You compile the code/amalgamation with SQLITE_EXTRA_INIT defined to the name of a function that takes a NULL parameter and returns an SQLITE error code (SQLITE_OK, SQLITE_ERROR, etc).  This function is called *after* the SQLite3 core is initialized.  It does additional initialization of the library.

For example, to autoload some extensions on every connection you can append the code for the extension to the amalgamation sqlite3.c file and then append you EXTRA_INIT function to add the extension init procedure to the auto extension list:

-DSQLITE_EXTRA_INIT=coreinit

static int coreinit(const char* dummy)
(
    return sqlite3_auto_extension((void*)sqlite3_series_init);
}

If you append series.c to the amalgamation, and then the coreinit function, and compile with -DSQLITE_EXTRA_INIT=coreinit, then when SQLite3 is initialized the coreinit function will run after the initialization is complete and add the sqlite3_series_init function to the auto_extension list.  Then when any new connection is opened the generate_series extension will be registered on that connection.

Of course, the extensions and the coreinit function do not have to be part of the amalgamation compilation unit -- they can be in a separate file and statically linked.

--
The fact that there's a Highway to Hell but only a Stairway to Heaven says a lot about anticipated traffic volume.



_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users