Re: LSM1 extension

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: LSM1 extension

Jens Alfke-2

> On Aug 4, 2017, at 9:12 PM, Charles Leifer <[hidden email]> wrote:
>
> Right -- my question is still unanswered as to why the code was merged in.

It’s not uncommon in software engineering to put unfinished long-running feature work on the main development branch but disable it with a flag. (It was standard policy in one of the projects I worked on at Google.) This helps keep the mainline code and unfinished code from diverging, and it simplifies testing of the unfinished code.

I’m eager for LSM1 too, but I’m waiting for it to show up in the release notes :)

—Jens
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: LSM1 extension

curmudgeon
I keep reading about this LSM1 but can’t find anything about it beyond tickets. Any links?

From: Jens Alfke<mailto:[hidden email]>
Sent: 08 August 2017 18:04
To: SQLite mailing list<mailto:[hidden email]>
Subject: Re: [sqlite] LSM1 extension


> On Aug 4, 2017, at 9:12 PM, Charles Leifer <[hidden email]> wrote:
>
> Right -- my question is still unanswered as to why the code was merged in.

It’s not uncommon in software engineering to put unfinished long-running feature work on the main development branch but disable it with a flag. (It was standard policy in one of the projects I worked on at Google.) This helps keep the mainline code and unfinished code from diverging, and it simplifies testing of the unfinished code.

I’m eager for LSM1 too, but I’m waiting for it to show up in the release notes :)

—Jens
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: LSM1 extension

Richard Hipp-3
On 8/8/17, x <[hidden email]> wrote:
> I keep reading about this LSM1 but can’t find anything about it beyond
> tickets. Any links?

Baseline technology: https://en.wikipedia.org/wiki/Log-structured_merge-tree

No documentation yet on the LSM1 implementation in SQLite.

--
D. Richard Hipp
[hidden email]
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: LSM1 extension

curmudgeon
Thanks Richard.

From: Richard Hipp<mailto:[hidden email]>
Sent: 08 August 2017 18:47
To: SQLite mailing list<mailto:[hidden email]>
Subject: Re: [sqlite] LSM1 extension

On 8/8/17, x <[hidden email]> wrote:
> I keep reading about this LSM1 but can’t find anything about it beyond
> tickets. Any links?

Baseline technology: https://en.wikipedia.org/wiki/Log-structured_merge-tree

No documentation yet on the LSM1 implementation in SQLite.

--
D. Richard Hipp
[hidden email]
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: LSM1 extension

Charles Leifer
There's some information that may be of interest on the sqlite4 wiki:

* Design overview: https://sqlite.org/src4/doc/trunk/www/lsm.wiki
* User guide: https://sqlite.org/src4/doc/trunk/www/lsmusr.wiki
* API: https://sqlite.org/src4/doc/trunk/www/lsmapi.wiki

My interest in this particular feature stems from my having written a set
of python bindings to the sqlite4 implementation (to be used as a sort of
embedded ordered key/value storage engine). I've wondered about the
possibility of exposing these APIs as a virtual table, and so seeing this
extension really excited me. So obviously my interest is much greater than
that of the average SQLite user :)

On Tue, Aug 8, 2017 at 12:54 PM, x <[hidden email]> wrote:

> Thanks Richard.
>
> From: Richard Hipp<mailto:[hidden email]>
> Sent: 08 August 2017 18:47
> To: SQLite mailing list<mailto:[hidden email]>
> Subject: Re: [sqlite] LSM1 extension
>
> On 8/8/17, x <[hidden email]> wrote:
> > I keep reading about this LSM1 but can’t find anything about it beyond
> > tickets. Any links?
>
> Baseline technology: https://en.wikipedia.org/wiki/
> Log-structured_merge-tree
>
> No documentation yet on the LSM1 implementation in SQLite.
>
> --
> D. Richard Hipp
> [hidden email]
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: LSM1 extension

curmudgeon
Thanks Charles. Is sqlite4 available yet?

From: Charles Leifer<mailto:[hidden email]>
Sent: 08 August 2017 21:21
To: SQLite mailing list<mailto:[hidden email]>
Subject: Re: [sqlite] LSM1 extension

There's some information that may be of interest on the sqlite4 wiki:

* Design overview: https://sqlite.org/src4/doc/trunk/www/lsm.wiki
* User guide: https://sqlite.org/src4/doc/trunk/www/lsmusr.wiki
* API: https://sqlite.org/src4/doc/trunk/www/lsmapi.wiki

My interest in this particular feature stems from my having written a set
of python bindings to the sqlite4 implementation (to be used as a sort of
embedded ordered key/value storage engine). I've wondered about the
possibility of exposing these APIs as a virtual table, and so seeing this
extension really excited me. So obviously my interest is much greater than
that of the average SQLite user :)

On Tue, Aug 8, 2017 at 12:54 PM, x <[hidden email]> wrote:

> Thanks Richard.
>
> From: Richard Hipp<mailto:[hidden email]>
> Sent: 08 August 2017 18:47
> To: SQLite mailing list<mailto:[hidden email]>
> Subject: Re: [sqlite] LSM1 extension
>
> On 8/8/17, x <[hidden email]> wrote:
> > I keep reading about this LSM1 but can’t find anything about it beyond
> > tickets. Any links?
>
> Baseline technology: https://en.wikipedia.org/wiki/
> Log-structured_merge-tree
>
> No documentation yet on the LSM1 implementation in SQLite.
>
> --
> D. Richard Hipp
> [hidden email]
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: LSM1 extension

Charles Leifer
sqlite4's future is uncertain. It's not, as I understand, meant to replace
sqlite3 any time soon. I think it was more of a place to try out new ideas
for implementations.

On Wed, Aug 9, 2017 at 3:10 AM, x <[hidden email]> wrote:

> Thanks Charles. Is sqlite4 available yet?
>
> From: Charles Leifer<mailto:[hidden email]>
> Sent: 08 August 2017 21:21
> To: SQLite mailing list<mailto:[hidden email]>
> Subject: Re: [sqlite] LSM1 extension
>
> There's some information that may be of interest on the sqlite4 wiki:
>
> * Design overview: https://sqlite.org/src4/doc/trunk/www/lsm.wiki
> * User guide: https://sqlite.org/src4/doc/trunk/www/lsmusr.wiki
> * API: https://sqlite.org/src4/doc/trunk/www/lsmapi.wiki
>
> My interest in this particular feature stems from my having written a set
> of python bindings to the sqlite4 implementation (to be used as a sort of
> embedded ordered key/value storage engine). I've wondered about the
> possibility of exposing these APIs as a virtual table, and so seeing this
> extension really excited me. So obviously my interest is much greater than
> that of the average SQLite user :)
>
> On Tue, Aug 8, 2017 at 12:54 PM, x <[hidden email]> wrote:
>
> > Thanks Richard.
> >
> > From: Richard Hipp<mailto:[hidden email]>
> > Sent: 08 August 2017 18:47
> > To: SQLite mailing list<mailto:[hidden email]>
> > Subject: Re: [sqlite] LSM1 extension
> >
> > On 8/8/17, x <[hidden email]> wrote:
> > > I keep reading about this LSM1 but can’t find anything about it beyond
> > > tickets. Any links?
> >
> > Baseline technology: https://en.wikipedia.org/wiki/
> > Log-structured_merge-tree
> >
> > No documentation yet on the LSM1 implementation in SQLite.
> >
> > --
> > D. Richard Hipp
> > [hidden email]
> > _______________________________________________
> > sqlite-users mailing list
> > [hidden email]
> > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
> >
> > _______________________________________________
> > sqlite-users mailing list
> > [hidden email]
> > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
> >
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: LSM1 extension

Nico Williams
On Wed, Aug 09, 2017 at 08:26:51AM -0500, Charles Leifer wrote:
> sqlite4's future is uncertain. It's not, as I understand, meant to replace
> sqlite3 any time soon. I think it was more of a place to try out new ideas
> for implementations.

I must say though, SQLite4 is *brilliant*.  I do hope that it either
gets completed and replaces SQLite3, or has all its good ideas folded
into SQLite3.  The LSM1 work in SQLite3 seems to point towords the
latter.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: LSM1 extension

Jens Alfke-2

> On Aug 9, 2017, at 8:23 AM, Nico Williams <[hidden email]> wrote:
>
> I must say though, SQLite4 is *brilliant*.  I do hope that it either
> gets completed and replaces SQLite3, or has all its good ideas folded
> into SQLite3.  The LSM1 work in SQLite3 seems to point towords the
> latter.

One thing I’m unclear on: Will the LSM1 extension in SQLite3 be able to plug in at a low level (replacing the b-tree engine), so one can use it with all the existing features like relational tables and SQL queries; or will it provide only a basic key/value store API?

(Sorry if it’s off-topic to discuss future features here. I know there’s a development-focused list but I think I’d be way beyond my depth there. LSM1 is potentially very, very relevant to the project I’m working on, but that depends on how it ends up integrated, which is why I ask.)

—Jens
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: LSM1 extension

Richard Hipp-3
On 8/9/17, Jens Alfke <[hidden email]> wrote:
>
> One thing I’m unclear on: Will the LSM1 extension in SQLite3 be able to plug
> in at a low level (replacing the b-tree engine), so one can use it with all
> the existing features like relational tables and SQL queries; or will it
> provide only a basic key/value store API?

It is just a separate key/value store using a different file.
--
D. Richard Hipp
[hidden email]
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Loading...