Is there any memory leak in the normal routine?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Is there any memory leak in the normal routine?

liubin liu
A routine of sqlite3_prepare_v2() + sqlite3_step() + sqlite3_finalize() could contain leak.

It sound ridiculous. But the test code seems to say it. Or I used the sqlite3_* wrongly.



______code__________________________________________

#include <stdio.h>
#include <unistd.h>     // for usleep()
#include <sqlite3.h>


int multi_write (int j);

sqlite3 *db = NULL;

int main (void)
{
    int ret = -1;

    ret = sqlite3_open("test.db", &db);
    ret = sqlite3_exec(db,"CREATE TABLE data_his (id INTEGER PRIMARY KEY, d1 CHAR(16))", NULL,NULL,NULL);
    usleep (100000);


    int j=0;
    while (1)
    {
        multi_write (j++);
        usleep (2000000);
        printf ("   ----------- %d\n", j);
    }


    ret = sqlite3_close (db);
    return 0;
}


int multi_write (int j)
{
    int ret = -1;

    char *sql_f = "INSERT OR REPLACE INTO data_his VALUES (%d, %Q)";
    char *sql = NULL;

    sqlite3_stmt *p_stmt = NULL;


    ret = sqlite3_prepare_v2 (db, "BEGIN TRANSACTION", -1, &p_stmt, NULL);
    ret = sqlite3_step ( p_stmt );
    ret = sqlite3_finalize ( p_stmt );

    int i=0;
    for (i=0; i<100; i++)
    {
        sql = sqlite3_mprintf ( sql_f, j*100000 + i, "00000000000068FD");

        ret = sqlite3_prepare_v2 (db, sql, -1, &p_stmt, NULL );
        sqlite3_free ( sql );
        //printf ("sqlite3_prepare_v2(): %d, %s\n", ret, sqlite3_errmsg (db));

        ret = sqlite3_step ( p_stmt );
        //printf ("sqlite3_step():       %d, %s\n", ret, sqlite3_errmsg (db));

        ret = sqlite3_finalize ( p_stmt );
        //printf ("sqlite3_finalize():   %d, %s\n\n", ret, sqlite3_errmsg (db));
    }

    ret = sqlite3_prepare_v2 (db, "COMMIT TRANSACTION", -1, &p_stmt, NULL );
    ret = sqlite3_step ( p_stmt );
    ret = sqlite3_finalize ( p_stmt );


    return 0;
}
Reply | Threaded
Open this post in threaded view
|

Re: Is there any memory leak in the normal routine?

liubin liu
And I watch the the process's run by top.

At first, the memory statistics is:
PID     PPID   USER     STAT   VSZ    %MEM  %CPU  COMMAND
17731 15488  root      S        1104   5%      7%     ./sqlite3multiwrite

When the printf() prints the 150, the memory statistics is:
PID     PPID   USER     STAT   VSZ    %MEM  %CPU  COMMAND
17731 15488  root      S        1552   5%       7%     ./sqlite3multiwrite


It means that after 150 for-cycles, the memory used by sqlite3multiwrite increase from 1104KB to 1552KB.

What does it mean? memory leak or other thing?
Reply | Threaded
Open this post in threaded view
|

Re: Is there any memory leak in the normal routine?

Black, Michael (IS)
I confirmed your "memory leak".  What you're seeing is the page cache growing.  Not really a memory leak.  Default page cache size is 2000 and indeed if I just let it run it topped out at 5404 RES in top.
 
 
I added dmalloc to sqlite3 and found that if you let your program loop several times and politely exit there is no memory leak.
 
So I added some debug statements in sqlite3.c to the alloc and free sections.  It produced an output like this:
alloc 664 0x2ac150a61408
alloc 3224 0x2aaaaaab9008
alloc 360 0x2aaaaaaab008
alloc 1016 0x2aaaaaac7808
free 0x2aaaaaab9008
free 0x2ac150a61408
free 0x2aaaaaac7808
free 0x2aaaaaaab008

I ran just two iterations of your program and edited the log to remove everything except the iteration between 1 & 2.
 
"grep alloc log | wc -l" and "grep free log" show that the 1st loop through your program shows 811 allocs and 809 frees.  So there are two places where memory is not getting freed.
 
I then wrote a quick log analysis that showed
free not found for 0x2ac150a63808
free not found for 0x2aaaaaac7008

grep for those two addresses showed
 
alloc 1280 0x2ac150a63808
alloc 1280 0x2aaaaaac7008
 
There are only two 1280 alloc's in the log do I added a debug statement in sqlite3MemMalloc to trigger on 1280 bytes.
Stepping through the debug then showed me at this point (line#'s are approximate as I've added statements) for both alloc's that occur:
Breakpoint 1, sqlite3MemMalloc (nByte=1272) at sqlite3.c:12977
#0  sqlite3MemMalloc (nByte=1272) at sqlite3.c:12977
#1  0x0000000000405b17 in mallocWithAlarm (n=1272, pp=0x7fff655c1248) at sqlite3.c:16343
#2  0x0000000000405bbc in sqlite3Malloc (n=1272) at sqlite3.c:16371
#3  0x000000000040fc5c in pcache1Alloc (nByte=1272) at sqlite3.c:31163
#4  0x000000000040fd39 in pcache1AllocPage (pCache=0x2acd74d1eb90) at sqlite3.c:31197
#5  0x00000000004105ca in pcache1Fetch (p=0x2acd74d1eb90, iKey=1, createFlag=2) at sqlite3.c:31566
#6  0x000000000040f2f3 in sqlite3PcacheFetch (pCache=0x2acd74d288f0, pgno=1, createFlag=1,
    ppPage=0x7fff655c1400) at sqlite3.c:30645
#7  0x000000000041470b in sqlite3PagerAcquire (pPager=0x2acd74d28810, pgno=1, ppPage=0x7fff655c1400,
    noContent=0) at sqlite3.c:36020
#8  0x0000000000418492 in btreeGetPage (pBt=0x2acd74d29c10, pgno=1, ppPage=0x7fff655c1450, noContent=0)
    at sqlite3.c:40101
#9  0x00000000004193bc in lockBtree (pBt=0x2acd74d29c10) at sqlite3.c:40811
#10 0x00000000004199a2 in sqlite3BtreeBeginTrans (p=0x2acd74d1ee90, wrflag=0) at sqlite3.c:41061
#11 0x0000000000455de7 in sqlite3InitOne (db=0x2acd74d28c10, iDb=0, pzErrMsg=0x2acd74d28420)
    at sqlite3.c:79614
#12 0x000000000045629d in sqlite3Init (db=0x2acd74d28c10, pzErrMsg=0x2acd74d28420) at sqlite3.c:79784
#13 0x0000000000456384 in sqlite3ReadSchema (pParse=0x2acd74d28410) at sqlite3.c:79821
#14 0x0000000000442d4a in sqlite3StartTable (pParse=0x2acd74d28410, pName1=0x2aaaaaab90e8,
    pName2=0x2aaaaaab9108, isTemp=0, isView=0, isVirtual=0, noErr=0) at sqlite3.c:68162
#15 0x000000000046bf2e in yy_reduce (yypParser=0x2aaaaaab9010, yyruleno=26) at sqlite3.c:94009
#16 0x000000000046f67e in sqlite3Parser (yyp=0x2aaaaaab9010, yymajor=22, yyminor=
      {z = 0x47b36e "(id INTEGER PRIMARY KEY, d1 CHAR(16))", n = 1}, pParse=0x2acd74d28410)
    at sqlite3.c:95191
#17 0x000000000047037b in sqlite3RunParser (pParse=0x2acd74d28410,
    zSql=0x47b358 "CREATE TABLE data_his (id INTEGER PRIMARY KEY, d1 CHAR(16))", pzErrMsg=0x7fff655c1980)
    at sqlite3.c:96017
#18 0x00000000004566fd in sqlite3Prepare (db=0x2acd74d28c10,
    zSql=0x47b358 "CREATE TABLE data_his (id INTEGER PRIMARY KEY, d1 CHAR(16))", nBytes=-1,
    saveSqlFlag=0, pReprepare=0x0, ppStmt=0x7fff655c1ac8, pzTail=0x7fff655c1ad0) at sqlite3.c:79995
#19 0x0000000000456a2b in sqlite3LockAndPrepare (db=0x2acd74d28c10,
    zSql=0x47b358 "CREATE TABLE data_his (id INTEGER PRIMARY KEY, d1 CHAR(16))", nBytes=-1,
    saveSqlFlag=0, pOld=0x0, ppStmt=0x7fff655c1ac8, pzTail=0x7fff655c1ad0) at sqlite3.c:80090
#20 0x0000000000456b97 in sqlite3_prepare (db=0x2acd74d28c10,
    zSql=0x47b358 "CREATE TABLE data_his (id INTEGER PRIMARY KEY, d1 CHAR(16))", nBytes=-1,
    ppStmt=0x7fff655c1ac8, pzTail=0x7fff655c1ad0) at sqlite3.c:80153
#21 0x0000000000451bc7 in sqlite3_exec (db=0x2acd74d28c10,
    zSql=0x47b358 "CREATE TABLE data_his (id INTEGER PRIMARY KEY, d1 CHAR(16))", xCallback=0, pArg=0x0,
    pzErrMsg=0x0) at sqlite3.c:76835
#22 0x0000000000401d8a in main () at thread.c:16


 
Michael D. Black
Senior Scientist
Northrop Grumman Mission Systems
 

________________________________

From: [hidden email] on behalf of liubin liu
Sent: Sat 4/24/2010 1:58 AM
To: [hidden email]
Subject: Re: [sqlite] Is there any memory leak in the normal routine?




And I watch the the process's run by top.

At first, the memory statistics is:
PID     PPID   USER     STAT   VSZ    %MEM  %CPU  COMMAND
17731 15488  root      S        1104   5%      7%     ./sqlite3multiwrite

When the printf() prints the 150, the memory statistics is:
PID     PPID   USER     STAT   VSZ    %MEM  %CPU  COMMAND
17731 15488  root      S        1552   5%       7%     ./sqlite3multiwrite


It means that after 150 for-cycles, the memory used by sqlite3multiwrite
increase from 1104KB to 1552KB.

What does it mean? memory leak or other thing?

--
View this message in context: http://old.nabble.com/Is-there-any-memory-leak-in-the-normal-routine--tp28348648p28348725.html
Sent from the SQLite mailing list archive at Nabble.com.

_______________________________________________
sqlite-users mailing list
[hidden email]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users



_______________________________________________
sqlite-users mailing list
[hidden email]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Is there any memory leak in the normal routine?

liubin liu
This post was updated on .
Thank you very much!

I test my code for enough time, and the memory using indeed stop increases when the process's memory using is 3524 RES.

And I try to "PRAGMA default_cache_size=100". Then the memory is just 1324 and stop increase.




Black, Michael (IS) wrote
I confirmed your "memory leak".  What you're seeing is the page cache growing.  Not really a memory leak.  Default page cache size is 2000 and indeed if I just let it run it topped out at 5404 RES in top.
 
 
I added dmalloc to sqlite3 and found that if you let your program loop several times and politely exit there is no memory leak.
 
So I added some debug statements in sqlite3.c to the alloc and free sections.  It produced an output like this:
alloc 664 0x2ac150a61408
alloc 3224 0x2aaaaaab9008
alloc 360 0x2aaaaaaab008
alloc 1016 0x2aaaaaac7808
free 0x2aaaaaab9008
free 0x2ac150a61408
free 0x2aaaaaac7808
free 0x2aaaaaaab008

I ran just two iterations of your program and edited the log to remove everything except the iteration between 1 & 2.
 
"grep alloc log | wc -l" and "grep free log" show that the 1st loop through your program shows 811 allocs and 809 frees.  So there are two places where memory is not getting freed.
 
I then wrote a quick log analysis that showed
free not found for 0x2ac150a63808
free not found for 0x2aaaaaac7008

grep for those two addresses showed
 
alloc 1280 0x2ac150a63808
alloc 1280 0x2aaaaaac7008
 
There are only two 1280 alloc's in the log do I added a debug statement in sqlite3MemMalloc to trigger on 1280 bytes.
Stepping through the debug then showed me at this point (line#'s are approximate as I've added statements) for both alloc's that occur:
Breakpoint 1, sqlite3MemMalloc (nByte=1272) at sqlite3.c:12977
#0  sqlite3MemMalloc (nByte=1272) at sqlite3.c:12977
#1  0x0000000000405b17 in mallocWithAlarm (n=1272, pp=0x7fff655c1248) at sqlite3.c:16343
#2  0x0000000000405bbc in sqlite3Malloc (n=1272) at sqlite3.c:16371
#3  0x000000000040fc5c in pcache1Alloc (nByte=1272) at sqlite3.c:31163
#4  0x000000000040fd39 in pcache1AllocPage (pCache=0x2acd74d1eb90) at sqlite3.c:31197
#5  0x00000000004105ca in pcache1Fetch (p=0x2acd74d1eb90, iKey=1, createFlag=2) at sqlite3.c:31566
#6  0x000000000040f2f3 in sqlite3PcacheFetch (pCache=0x2acd74d288f0, pgno=1, createFlag=1,
    ppPage=0x7fff655c1400) at sqlite3.c:30645
#7  0x000000000041470b in sqlite3PagerAcquire (pPager=0x2acd74d28810, pgno=1, ppPage=0x7fff655c1400,
    noContent=0) at sqlite3.c:36020
#8  0x0000000000418492 in btreeGetPage (pBt=0x2acd74d29c10, pgno=1, ppPage=0x7fff655c1450, noContent=0)
    at sqlite3.c:40101
#9  0x00000000004193bc in lockBtree (pBt=0x2acd74d29c10) at sqlite3.c:40811
#10 0x00000000004199a2 in sqlite3BtreeBeginTrans (p=0x2acd74d1ee90, wrflag=0) at sqlite3.c:41061
#11 0x0000000000455de7 in sqlite3InitOne (db=0x2acd74d28c10, iDb=0, pzErrMsg=0x2acd74d28420)
    at sqlite3.c:79614
#12 0x000000000045629d in sqlite3Init (db=0x2acd74d28c10, pzErrMsg=0x2acd74d28420) at sqlite3.c:79784
#13 0x0000000000456384 in sqlite3ReadSchema (pParse=0x2acd74d28410) at sqlite3.c:79821
#14 0x0000000000442d4a in sqlite3StartTable (pParse=0x2acd74d28410, pName1=0x2aaaaaab90e8,
    pName2=0x2aaaaaab9108, isTemp=0, isView=0, isVirtual=0, noErr=0) at sqlite3.c:68162
#15 0x000000000046bf2e in yy_reduce (yypParser=0x2aaaaaab9010, yyruleno=26) at sqlite3.c:94009
#16 0x000000000046f67e in sqlite3Parser (yyp=0x2aaaaaab9010, yymajor=22, yyminor=
      {z = 0x47b36e "(id INTEGER PRIMARY KEY, d1 CHAR(16))", n = 1}, pParse=0x2acd74d28410)
    at sqlite3.c:95191
#17 0x000000000047037b in sqlite3RunParser (pParse=0x2acd74d28410,
    zSql=0x47b358 "CREATE TABLE data_his (id INTEGER PRIMARY KEY, d1 CHAR(16))", pzErrMsg=0x7fff655c1980)
    at sqlite3.c:96017
#18 0x00000000004566fd in sqlite3Prepare (db=0x2acd74d28c10,
    zSql=0x47b358 "CREATE TABLE data_his (id INTEGER PRIMARY KEY, d1 CHAR(16))", nBytes=-1,
    saveSqlFlag=0, pReprepare=0x0, ppStmt=0x7fff655c1ac8, pzTail=0x7fff655c1ad0) at sqlite3.c:79995
#19 0x0000000000456a2b in sqlite3LockAndPrepare (db=0x2acd74d28c10,
    zSql=0x47b358 "CREATE TABLE data_his (id INTEGER PRIMARY KEY, d1 CHAR(16))", nBytes=-1,
    saveSqlFlag=0, pOld=0x0, ppStmt=0x7fff655c1ac8, pzTail=0x7fff655c1ad0) at sqlite3.c:80090
#20 0x0000000000456b97 in sqlite3_prepare (db=0x2acd74d28c10,
    zSql=0x47b358 "CREATE TABLE data_his (id INTEGER PRIMARY KEY, d1 CHAR(16))", nBytes=-1,
    ppStmt=0x7fff655c1ac8, pzTail=0x7fff655c1ad0) at sqlite3.c:80153
#21 0x0000000000451bc7 in sqlite3_exec (db=0x2acd74d28c10,
    zSql=0x47b358 "CREATE TABLE data_his (id INTEGER PRIMARY KEY, d1 CHAR(16))", xCallback=0, pArg=0x0,
    pzErrMsg=0x0) at sqlite3.c:76835
#22 0x0000000000401d8a in main () at thread.c:16


 
Michael D. Black
Senior Scientist
Northrop Grumman Mission Systems
 

________________________________

From: sqlite-users-bounces@sqlite.org on behalf of liubin liu
Sent: Sat 4/24/2010 1:58 AM
To: sqlite-users@sqlite.org
Subject: Re: [sqlite] Is there any memory leak in the normal routine?




And I watch the the process's run by top.

At first, the memory statistics is:
PID     PPID   USER     STAT   VSZ    %MEM  %CPU  COMMAND
17731 15488  root      S        1104   5%      7%     ./sqlite3multiwrite

When the printf() prints the 150, the memory statistics is:
PID     PPID   USER     STAT   VSZ    %MEM  %CPU  COMMAND
17731 15488  root      S        1552   5%       7%     ./sqlite3multiwrite


It means that after 150 for-cycles, the memory used by sqlite3multiwrite
increase from 1104KB to 1552KB.

What does it mean? memory leak or other thing?

--
View this message in context: http://old.nabble.com/Is-there-any-memory-leak-in-the-normal-routine--tp28348648p28348725.html
Sent from the SQLite mailing list archive at Nabble.com.

_______________________________________________
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users



_______________________________________________
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users