LSM Extension Performance (from sqlite3 tree)

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

LSM Extension Performance (from sqlite3 tree)

Amirouche Boubekki
Hello,


I am considering using lsm extension for a project. I a did a few benchmark
with the following configuration:

LSM_CONFIG_AUTOFLUSH 1048576
LSM_CONFIG_BLOCK_SIZE 65536
LSM_CONFIG_AUTOWORK 1
LSM_CONFIG_MMAP 0
LSM_CONFIG_MULTIPLE_PROCESS 0

I am trying to load 4.5GB of data. The database errors with BUSY error at
some point.

Also, the memory consumption grows unbound even if I diseabled MMAP.

Here is some stats:

$ ls -larh wt/foob.ar.lsm.sqlite*

-rw-r--r-- 1 amirouche amirouche 618M May 16 13:39 wt/foob.ar.lsm.sqlite-log
-rw-r--r-- 1 amirouche amirouche 468M May 16 13:39 wt/foob.ar.lsm.sqlite

And I attached a memory plot.

opening and closing the database at each transaction help, but still the
memory grows.

What I am doing wrong?
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: LSM Extension Performance (from sqlite3 tree)

Dan Kennedy-4

On 16/5/62 18:47, Amirouche Boubekki wrote:

> Hello,
>
>
> I am considering using lsm extension for a project. I a did a few benchmark
> with the following configuration:
>
> LSM_CONFIG_AUTOFLUSH 1048576
> LSM_CONFIG_BLOCK_SIZE 65536
> LSM_CONFIG_AUTOWORK 1
> LSM_CONFIG_MMAP 0
> LSM_CONFIG_MULTIPLE_PROCESS 0
>
> I am trying to load 4.5GB of data. The database errors with BUSY error at
> some point.
>
> Also, the memory consumption grows unbound even if I diseabled MMAP.
>
> Here is some stats:
>
> $ ls -larh wt/foob.ar.lsm.sqlite*
>
> -rw-r--r-- 1 amirouche amirouche 618M May 16 13:39 wt/foob.ar.lsm.sqlite-log
> -rw-r--r-- 1 amirouche amirouche 468M May 16 13:39 wt/foob.ar.lsm.sqlite
>
> And I attached a memory plot.
>
> opening and closing the database at each transaction help, but still the
> memory grows.
>
> What I am doing wrong?

The mailing list has discarded your attachment. But what we need most to
try to diagnose the memory leak is the code you are using. If we can
actually run it and reproduce the memory leak, so much the better.

Dan.



> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: LSM Extension Performance (from sqlite3 tree)

Amirouche Boubekki
Le jeu. 16 mai 2019 à 15:46, Dan Kennedy <[hidden email]> a écrit :

>
> On 16/5/62 18:47, Amirouche Boubekki wrote:
> > Hello,
> >
> >
> > I am considering using lsm extension for a project. I a did a few
> benchmark
> > with the following configuration:
> >
> > LSM_CONFIG_AUTOFLUSH 1048576
> > LSM_CONFIG_BLOCK_SIZE 65536
> > LSM_CONFIG_AUTOWORK 1
> > LSM_CONFIG_MMAP 0
> > LSM_CONFIG_MULTIPLE_PROCESS 0
> >
> > I am trying to load 4.5GB of data. The database errors with BUSY error at
> > some point.
> >
> > Also, the memory consumption grows unbound even if I diseabled MMAP.
> >
> > Here is some stats:
> >
> > $ ls -larh wt/foob.ar.lsm.sqlite*
> >
> > -rw-r--r-- 1 amirouche amirouche 618M May 16 13:39
> wt/foob.ar.lsm.sqlite-log
> > -rw-r--r-- 1 amirouche amirouche 468M May 16 13:39 wt/foob.ar.lsm.sqlite
> >
> > And I attached a memory plot.
> >
> > opening and closing the database at each transaction help, but still the
> > memory grows.
> >
> > What I am doing wrong?
>
> The mailing list has discarded your attachment. But what we need most to
> try to diagnose the memory leak is the code you are using. If we can
> actually run it and reproduce the memory leak, so much the better.
>
> Dan.
>

I have several layers between the database and the the actual code. It is
not
easy to run, also it rely Chez Scheme (which is available in Ubuntu 18.04,
anyway it is easy to compile and run).

I can create a small program that read the key/value pairs from a file that
you
would be able to re-use in another program written in C for instance.

Would that work for you?
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: LSM Extension Performance (from sqlite3 tree)

Simon Slavin-3
On 16 May 2019, at 3:51pm, Amirouche Boubekki <[hidden email]> wrote:

> I can create a small program that read the key/value pairs from a file that you would be able to re-use in another program written in C for instance.

Can you create a small program which shows the problem you are complaining about ?  Preferably in C.  That would be very useful.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: LSM Extension Performance (from sqlite3 tree)

Amirouche Boubekki
In reply to this post by Amirouche Boubekki
Le jeu. 16 mai 2019 à 13:47, Amirouche Boubekki
<[hidden email]> a écrit :

>
> Hello,
>
>
> I am considering using lsm extension for a project. I a did a few benchmark with the following configuration:
>
> LSM_CONFIG_AUTOFLUSH 1048576
> LSM_CONFIG_BLOCK_SIZE 65536
> LSM_CONFIG_AUTOWORK 1
> LSM_CONFIG_MMAP 0
> LSM_CONFIG_MULTIPLE_PROCESS 0
>
> I am trying to load 4.5GB of data. The database errors with BUSY error at some point.
>
> Also, the memory consumption grows unbound even if I diseabled MMAP.
>
> Here is some stats:
>
> $ ls -larh wt/foob.ar.lsm.sqlite*
>
> -rw-r--r-- 1 amirouche amirouche 618M May 16 13:39 wt/foob.ar.lsm.sqlite-log
> -rw-r--r-- 1 amirouche amirouche 468M May 16 13:39 wt/foob.ar.lsm.sqlite
>
> And I attached a memory plot.
>
> opening and closing the database at each transaction help, but still the memory grows.
>
> What I am doing wrong?

It is probably because I was inserting bytes taken from /dev/urandom.

The code is at https://git.sr.ht/~amz3/chez-sqlite-lsm/tree/master/bench.scm



--
Amirouche ~ amz3 ~ https://hyper.dev
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users