LSM extension unusual memory growth?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

LSM extension unusual memory growth?

Josu Diaz de Arcaya
Hi all,

I have been giving the LSM extension a try and I have a concern
regarding memory consumption. It seems as as long as you keep the
connection open and insert and delete
entries memory keeps growing without an apparent roof. I have made a
very simple program that reproduces this.

#include "lsm.h"

#include <stdio.h>
#include <stdint.h>
#include <stdlib.h>
#include <signal.h>

int j=-1;

void generateKey(char *key);

int main() {

        lsm_db *db;
        lsm_new(lsm_default_env(), &db);
        lsm_open(db, "test.db");

        char key[5], val[5];
        generateKey(val);

        lsm_cursor *csr;
        // delete and insert 400M elements
        for (j=0; j<400000000;j++) {
                generateKey(key);
                lsm_delete(db, key, 3);
                generateKey(key);
                lsm_insert(db, key, 3, val, 3);
        }

        lsm_close(db);
}

void generateKey(char *key) {
        key[0] = ( rand() % 40 ) + 45;
        key[1] = ( rand() % 40 ) + 45;
        key[2] = ( rand() % 40 ) + 45;
        key[3] = '\0';
This program deletes and insert entries into the database, the maximum
amount of entries is 40^3 but it stabilizes around half that much.
However, memory keeps growing
as shown in the next graph, which corresponds to a 15 hour run.
(Forgive me if the quality is not too good in the browser, if you copy
and paste it in a text editor it shows a constant slope)

--------------------------------------------------------------------------------
Command:            ./test
Massif arguments:   --stacks=yes
ms_print arguments: massif.out.11718
--------------------------------------------------------------------------------

    MB
10.48^                                                                       :
     |                                                                    @@@#
     |                                                                 @@@@  #
     |                                                             :@@@@@ @  #
     |                                                          @@@:@ @@@ @  #
     |                                                      @@@@@@@:@ @@@ @  #
     |                                                   @@@@ @ @@@:@ @@@ @  #
     |                                               :@@@@ @@ @ @@@:@ @@@ @  #
     |                                            ::::@@ @ @@ @ @@@:@ @@@ @  #
     |                                        @@@@:: :@@ @ @@ @ @@@:@ @@@ @  #
     |                                     @@@@ @ :: :@@ @ @@ @ @@@:@ @@@ @  #
     |                                @@@@@@ @@ @ :: :@@ @ @@ @ @@@:@ @@@ @  #
     |                              @@@@ @ @ @@ @ :: :@@ @ @@ @ @@@:@ @@@ @  #
     |                         @::::@@@@ @ @ @@ @ :: :@@ @ @@ @ @@@:@ @@@ @  #
     |                      @@:@: : @@@@ @ @ @@ @ :: :@@ @ @@ @ @@@:@ @@@ @  #
     |                 @@@@:@@:@: : @@@@ @ @ @@ @ :: :@@ @ @@ @ @@@:@ @@@ @  #
     |              @@:@ @ :@@:@: : @@@@ @ @ @@ @ :: :@@ @ @@ @ @@@:@ @@@ @  #
     |           ::@@ :@ @ :@@:@: : @@@@ @ @ @@ @ :: :@@ @ @@ @ @@@:@ @@@ @  #
     |       ::::: @@ :@ @ :@@:@: : @@@@ @ @ @@ @ :: :@@ @ @@ @ @@@:@ @@@ @  #
     |   :::::: :: @@ :@ @ :@@:@: : @@@@ @ @ @@ @ :: :@@ @ @@ @ @@@:@ @@@ @  #
   0 +----------------------------------------------------------------------->Ti
     0                                                                   8.616

I've run this test with valgrind's massif tool. I've made longer tests
but the behaviour looks the same. It looks to me that there is
something odd going on with memory,
but I may be missing something.

Is this a problem with LSM extension itself or am I missing some
important point in here?

Thanks!
Josu
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users