Quantcast

QUERY: Usage of malloc and free in SQLite

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

QUERY: Usage of malloc and free in SQLite

Subramaniyan, Ganesan (G.)
Hi,
We are facing dynamic memory pool corruption issue while using SQLite. Based on our investigation we suspect SQLite freeing memory bypassing the memory management API'S registered as below.

We have registered the memory allocation routines as below.
static const sqlite3_mem_methods mqxmem = {
        mqxMalloc,
        mqxFree,
        mqxRealloc,
        mqxSize,
        mqxRoundup,
        mqxInit,
        mqxShutdown,
        NULL
    };
sqlite3_config(SQLITE_CONFIG_MALLOC, &mqxmem);

Despite this, is there any sort of configurations by which SQLite might still use its own or standard library APIs to allocate/reallocate or free the memory, bypassing the memory management APIs registered above?
We are using SQLite version 3.7.10 running on Freescale MQX hosted on ARM A5 Cortex processor.
We have enabled SQLITE_ENABLE_MEMORY_MANAGEMENT configuration and using sqlite3_release_memory()<https://www.sqlite.org/c3ref/release_memory.html> very often, after every fetch/write operations.

Thanks & Regards,
Ganesan.S
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: QUERY: Usage of malloc and free in SQLite

Dan Kennedy-4
On 03/23/2017 10:00 PM, Subramaniyan, Ganesan (G.) wrote:

> Hi,
> We are facing dynamic memory pool corruption issue while using SQLite. Based on our investigation we suspect SQLite freeing memory bypassing the memory management API'S registered as below.
>
> We have registered the memory allocation routines as below.
> static const sqlite3_mem_methods mqxmem = {
>          mqxMalloc,
>          mqxFree,
>          mqxRealloc,
>          mqxSize,
>          mqxRoundup,
>          mqxInit,
>          mqxShutdown,
>          NULL
>      };
> sqlite3_config(SQLITE_CONFIG_MALLOC, &mqxmem);
>
> Despite this, is there any sort of configurations by which SQLite might still use its own or standard library APIs to allocate/reallocate or free the memory, bypassing the memory management APIs registered above?


No. Following a successful call to sqlite3_config() SQLite allocates and
frees memory using the configured routines exclusively[1]. There are no
direct calls to malloc()/free() or similar in the library.

Are you checking the return value of sqlite3_config()? If
sqlite3_initialize() or any other sqlite3_*() function has already been
called when sqlite3_config() is invoked it will fail and the memory
allocation routines will not be configured.

   https://sqlite.org/c3ref/config.html

What symptoms are you seeing that suggest SQLite is bypassing your
memory allocation functions?

Dan.

[1] It may also make use of static buffers supplied by calls to
sqlite3_config(SQLITE_CONFIG_HEAP), CONFIG_PAGECACHE or CONFIG_SCRATCH,
but I'm guessing you have made no such calls. Most apps do not.




> We are using SQLite version 3.7.10 running on Freescale MQX hosted on ARM A5 Cortex processor.
> We have enabled SQLITE_ENABLE_MEMORY_MANAGEMENT configuration and using sqlite3_release_memory()<https://www.sqlite.org/c3ref/release_memory.html> very often, after every fetch/write operations.
>
> Thanks & Regards,
> Ganesan.S
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: QUERY: Usage of malloc and free in SQLite

J Decker
On Thu, Mar 23, 2017 at 9:01 AM, Dan Kennedy <[hidden email]> wrote:

> On 03/23/2017 10:00 PM, Subramaniyan, Ganesan (G.) wrote:
>
>> Hi,
>> We are facing dynamic memory pool corruption issue while using SQLite.
>> Based on our investigation we suspect SQLite freeing memory bypassing the
>> memory management API'S registered as below.
>>
>> We have registered the memory allocation routines as below.
>> static const sqlite3_mem_methods mqxmem = {
>>          mqxMalloc,
>>          mqxFree,
>>          mqxRealloc,
>>          mqxSize,
>>          mqxRoundup,
>>          mqxInit,
>>          mqxShutdown,
>>          NULL
>>      };
>> sqlite3_config(SQLITE_CONFIG_MALLOC, &mqxmem);
>>
>> Despite this, is there any sort of configurations by which SQLite might
>> still use its own or standard library APIs to allocate/reallocate or free
>> the memory, bypassing the memory management APIs registered above?
>>
>
>
> No. Following a successful call to sqlite3_config() SQLite allocates and
> frees memory using the configured routines exclusively[1]. There are no
> direct calls to malloc()/free() or similar in the library.
>
> Are you checking the return value of sqlite3_config()? If
> sqlite3_initialize() or any other sqlite3_*() function has already been
> called when sqlite3_config() is invoked it will fail and the memory
> allocation routines will not be configured.
>
>   https://sqlite.org/c3ref/config.html
>
> What symptoms are you seeing that suggest SQLite is bypassing your memory
> allocation functions?
>

There is an internal check to see if sqlite has alerady 'initialized' the
config needs to be set before it's initialized.  There's a error thrown

void errorLogCallback(void *pArg, int iErrCode, const char *zMsg){
printf( "Sqlite3 Err: (%d) %s", iErrCode, zMsg);
}

sqlite3_config( SQLITE_CONFIG_LOG, errorLogCallback, 0);

that is sqlite3_config is called late it logs an 'invalid operation' sort
of message.

>
> Dan.
>
> [1] It may also make use of static buffers supplied by calls to
> sqlite3_config(SQLITE_CONFIG_HEAP), CONFIG_PAGECACHE or CONFIG_SCRATCH,
> but I'm guessing you have made no such calls. Most apps do not.
>
>
>
>
>
> We are using SQLite version 3.7.10 running on Freescale MQX hosted on ARM
>> A5 Cortex processor.
>> We have enabled SQLITE_ENABLE_MEMORY_MANAGEMENT configuration and using
>> sqlite3_release_memory()<https://www.sqlite.org/c3ref/release_memory.html>
>> very often, after every fetch/write operations.
>>
>> Thanks & Regards,
>> Ganesan.S
>> _______________________________________________
>> sqlite-users mailing list
>> [hidden email]
>> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>>
>
>
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: QUERY: Usage of malloc and free in SQLite

Anthrathodiyil, Sabeel (S.)
In reply to this post by Dan Kennedy-4
Hi Dan,
Custom memory management APIs registered are being used by SQLite. We suspect SQLite, in certain cases is freeing allocated memory, not using the registered API to free.

As part of our investigation, we are maintaining the starting address of the memory allocated and size allocated, and removed from list once it is freed via registered memory management API mqxFree. But we find the same address being allocated again for a subsequent memory allocation/reallocation request which was not freed using mqxFree. This leads us to suspect allocated memory is being freed(not through the registered API) and same address is being made available for next allocation.

Thanks,
Sabeel

-----Original Message-----
From: sqlite-users [mailto:[hidden email]] On Behalf Of Dan Kennedy
Sent: Thursday, March 23, 2017 9:32 PM
To: [hidden email]
Subject: Re: [sqlite] QUERY: Usage of malloc and free in SQLite

On 03/23/2017 10:00 PM, Subramaniyan, Ganesan (G.) wrote:

> Hi,
> We are facing dynamic memory pool corruption issue while using SQLite. Based on our investigation we suspect SQLite freeing memory bypassing the memory management API'S registered as below.
>
> We have registered the memory allocation routines as below.
> static const sqlite3_mem_methods mqxmem = {
>          mqxMalloc,
>          mqxFree,
>          mqxRealloc,
>          mqxSize,
>          mqxRoundup,
>          mqxInit,
>          mqxShutdown,
>          NULL
>      };
> sqlite3_config(SQLITE_CONFIG_MALLOC, &mqxmem);
>
> Despite this, is there any sort of configurations by which SQLite might still use its own or standard library APIs to allocate/reallocate or free the memory, bypassing the memory management APIs registered above?


No. Following a successful call to sqlite3_config() SQLite allocates and frees memory using the configured routines exclusively[1]. There are no direct calls to malloc()/free() or similar in the library.

Are you checking the return value of sqlite3_config()? If
sqlite3_initialize() or any other sqlite3_*() function has already been called when sqlite3_config() is invoked it will fail and the memory allocation routines will not be configured.

   https://sqlite.org/c3ref/config.html

What symptoms are you seeing that suggest SQLite is bypassing your memory allocation functions?

Dan.

[1] It may also make use of static buffers supplied by calls to sqlite3_config(SQLITE_CONFIG_HEAP), CONFIG_PAGECACHE or CONFIG_SCRATCH, but I'm guessing you have made no such calls. Most apps do not.




> We are using SQLite version 3.7.10 running on Freescale MQX hosted on ARM A5 Cortex processor.
> We have enabled SQLITE_ENABLE_MEMORY_MANAGEMENT configuration and using sqlite3_release_memory()<https://www.sqlite.org/c3ref/release_memory.html> very often, after every fetch/write operations.
>
> Thanks & Regards,
> Ganesan.S
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: QUERY: Usage of malloc and free in SQLite

Hick Gunter
If I Interpret your text correctly, you are claiming that your mpxMalloc/mpxRealloc is issuing the same address twice, without mpxFree being called in between?

Are you tracking mpxRealloc calls correctly? If the area pointed to is not large enough, a new area will be allocated, the data copied and the old area freed.

Otherwise, it would imply that your memory allocation is faulty because it is allocating the same memory twice without an intervening free.


-----Ursprüngliche Nachricht-----
Von: sqlite-users [mailto:[hidden email]] Im Auftrag von Anthrathodiyil, Sabeel (S.)
Gesendet: Freitag, 24. März 2017 12:19
An: SQLite mailing list <[hidden email]>
Cc: Subramaniyan, Ganesan (G.) <[hidden email]>; Natanam, Karthigeyan (K.) <[hidden email]>
Betreff: Re: [sqlite] QUERY: Usage of malloc and free in SQLite

Hi Dan,
Custom memory management APIs registered are being used by SQLite. We suspect SQLite, in certain cases is freeing allocated memory, not using the registered API to free.

As part of our investigation, we are maintaining the starting address of the memory allocated and size allocated, and removed from list once it is freed via registered memory management API mqxFree. But we find the same address being allocated again for a subsequent memory allocation/reallocation request which was not freed using mqxFree. This leads us to suspect allocated memory is being freed(not through the registered API) and same address is being made available for next allocation.

Thanks,
Sabeel

-----Original Message-----
From: sqlite-users [mailto:[hidden email]] On Behalf Of Dan Kennedy
Sent: Thursday, March 23, 2017 9:32 PM
To: [hidden email]
Subject: Re: [sqlite] QUERY: Usage of malloc and free in SQLite

On 03/23/2017 10:00 PM, Subramaniyan, Ganesan (G.) wrote:

> Hi,
> We are facing dynamic memory pool corruption issue while using SQLite. Based on our investigation we suspect SQLite freeing memory bypassing the memory management API'S registered as below.
>
> We have registered the memory allocation routines as below.
> static const sqlite3_mem_methods mqxmem = {
>          mqxMalloc,
>          mqxFree,
>          mqxRealloc,
>          mqxSize,
>          mqxRoundup,
>          mqxInit,
>          mqxShutdown,
>          NULL
>      };
> sqlite3_config(SQLITE_CONFIG_MALLOC, &mqxmem);
>
> Despite this, is there any sort of configurations by which SQLite might still use its own or standard library APIs to allocate/reallocate or free the memory, bypassing the memory management APIs registered above?


No. Following a successful call to sqlite3_config() SQLite allocates and frees memory using the configured routines exclusively[1]. There are no direct calls to malloc()/free() or similar in the library.

Are you checking the return value of sqlite3_config()? If
sqlite3_initialize() or any other sqlite3_*() function has already been called when sqlite3_config() is invoked it will fail and the memory allocation routines will not be configured.

   https://sqlite.org/c3ref/config.html

What symptoms are you seeing that suggest SQLite is bypassing your memory allocation functions?

Dan.

[1] It may also make use of static buffers supplied by calls to sqlite3_config(SQLITE_CONFIG_HEAP), CONFIG_PAGECACHE or CONFIG_SCRATCH, but I'm guessing you have made no such calls. Most apps do not.




> We are using SQLite version 3.7.10 running on Freescale MQX hosted on ARM A5 Cortex processor.
> We have enabled SQLITE_ENABLE_MEMORY_MANAGEMENT configuration and using sqlite3_release_memory()<https://www.sqlite.org/c3ref/release_memory.html> very often, after every fetch/write operations.
>
> Thanks & Regards,
> Ganesan.S
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


___________________________________________
 Gunter Hick
Software Engineer
Scientific Games International GmbH
FN 157284 a, HG Wien
Klitschgasse 2-4, A-1130 Vienna, Austria
Tel: +43 1 80100 0
E-Mail: [hidden email]

This communication (including any attachments) is intended for the use of the intended recipient(s) only and may contain information that is confidential, privileged or legally protected. Any unauthorized use or dissemination of this communication is strictly prohibited. If you have received this communication in error, please immediately notify the sender by return e-mail message and delete all copies of the original communication. Thank you for your cooperation.


_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: QUERY: Usage of malloc and free in SQLite

Graham Holden
In reply to this post by Subramaniyan, Ganesan (G.)
I've no idea of the internals of SQLite, but two things come to mind:
1. Can your custom allocation routines return the same address in the case of a realloc() call that is either smaller than the original allocation, or (if applicable) is larger but still fits in the original "memory block"? This might explain the duplicate entries you're seeing.
2. If a realloc() call needs a new block of memory to satisfy it, your allocators must, presumably, be freeing the original block. Are those "internal" frees being tracked? If so, the freed block may be returned by a subsequent allocation, without (apparently) being freed.
Graham
Sent from my Samsung Galaxy S7 - powered by Three
-------- Original message --------From: "Anthrathodiyil, Sabeel (S.)" <[hidden email]> Date: 24/03/2017  11:18  (GMT+00:00) To: SQLite mailing list <[hidden email]> Cc: "Subramaniyan, Ganesan (G.)" <[hidden email]>, "Natanam, Karthigeyan (K.)" <[hidden email]> Subject: Re: [sqlite] QUERY: Usage of malloc and free in SQLite
Hi Dan,
Custom memory management APIs registered are being used by SQLite. We suspect SQLite, in certain cases is freeing allocated memory, not using the registered API to free.

As part of our investigation, we are maintaining the starting address of the memory allocated and size allocated, and removed from list once it is freed via registered memory management API mqxFree. But we find the same address being allocated again for a subsequent memory allocation/reallocation request which was not freed using mqxFree. This leads us to suspect allocated memory is being freed(not through the registered API) and same address is being made available for next allocation.

Thanks,
Sabeel

-----Original Message-----
From: sqlite-users [mailto:[hidden email]] On Behalf Of Dan Kennedy
Sent: Thursday, March 23, 2017 9:32 PM
To: [hidden email]
Subject: Re: [sqlite] QUERY: Usage of malloc and free in SQLite

On 03/23/2017 10:00 PM, Subramaniyan, Ganesan (G.) wrote:

> Hi,
> We are facing dynamic memory pool corruption issue while using SQLite. Based on our investigation we suspect SQLite freeing memory bypassing the memory management API'S registered as below.
>
> We have registered the memory allocation routines as below.
> static const sqlite3_mem_methods mqxmem = {
>          mqxMalloc,
>          mqxFree,
>          mqxRealloc,
>          mqxSize,
>          mqxRoundup,
>          mqxInit,
>          mqxShutdown,
>          NULL
>      };
> sqlite3_config(SQLITE_CONFIG_MALLOC, &mqxmem);
>
> Despite this, is there any sort of configurations by which SQLite might still use its own or standard library APIs to allocate/reallocate or free the memory, bypassing the memory management APIs registered above?


No. Following a successful call to sqlite3_config() SQLite allocates and frees memory using the configured routines exclusively[1]. There are no direct calls to malloc()/free() or similar in the library.

Are you checking the return value of sqlite3_config()? If
sqlite3_initialize() or any other sqlite3_*() function has already been called when sqlite3_config() is invoked it will fail and the memory allocation routines will not be configured.

   https://sqlite.org/c3ref/config.html

What symptoms are you seeing that suggest SQLite is bypassing your memory allocation functions?

Dan.

[1] It may also make use of static buffers supplied by calls to sqlite3_config(SQLITE_CONFIG_HEAP), CONFIG_PAGECACHE or CONFIG_SCRATCH, but I'm guessing you have made no such calls. Most apps do not.




> We are using SQLite version 3.7.10 running on Freescale MQX hosted on ARM A5 Cortex processor.
> We have enabled SQLITE_ENABLE_MEMORY_MANAGEMENT configuration and using sqlite3_release_memory()<https://www.sqlite.org/c3ref/release_memory.html> very often, after every fetch/write operations.
>
> Thanks & Regards,
> Ganesan.S
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Loading...