BUG in 3.8.8.3 and pre-release: Wrong ERROR_DISK_FULL writing a blob on Windows

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

BUG in 3.8.8.3 and pre-release: Wrong ERROR_DISK_FULL writing a blob on Windows

ales@elxala.de
Hi, this is actually a Ticket, the good new is that is already pre-analyzed
and probably also the cause has been found.
I hope this is the right place to put this

TITLE: Wrong ERROR_DISK_FULL writing a blob on Windows

DESCRIPTION:

    Having a sqlite database about 4.1 GB
    writing blobs in some table, for instance using a statement like
       INSERT INTO inodeCargo VALUES (1000, readfile ("ScreenShot.png"));

    an error ERROR_DISK_FULL can be through wrongly (under certain
conditions)

    Windows 7 64-bit

REPRODUCIBLE:
    When the conditions are met, always reproducible


CAUSE :

    windows function WriteFile return error 112 (ERROR_DISK_FULL) when
trying to write
    1024 bytes using a offset = 4294966272 = 0xFFFFFC00 = FFFFFFFF - 2023

    This was observed using the default mode (OVERLAPPED) although a
test compiling
    the code with
       #define SQLITE_WIN32_NO_OVERLAPPED
    at the begining also failed.

PATCH:

    The patch (see sqlite3_modified.c) work in my case, but obviously cannot
    fix the mode NO_OVERLAPPED (it does not compile in this mode). Also
all procedures
    using the buggy function (if this result to be the cause) should be
reviewed.
    A more precise fix has to be developed by some expert in sqlite code.

ANALYSIS:

    A sqlite db used to store files came in the conditions described
    size of database file : 4.194.303 KB
    size of journal file : 8 KB
    size of the image (png file) : 4.417 KB

    SQL Query returning systematically ERROR_DISK_FULL:
       INSERT INTO inodeCargo VALUES (-24, 4522876, 0, readfile
("ScreenShot320.png"));

    (hard disk of the test with about 1 TB free space)

    Since the case was 100% reproducible I could debug the problem using
    sqlite-amalgamation-3080803.zip (ver 3.8.8.3)
    I tried sqlite-amalgamation-201503091040.zip as well with the same
    result, so the problem was also not fixed in this pre-release.

    In debug session I found that the error came from

       static int winWrite (...

    and it is given in the line

       if( !osWriteFile(pFile->h, aRem, nRem, &nWrite, &overlapped) ){

    which is a call to a windows api function WriteFile

    adding the extra debug messages into a file (see sqlite3_modified.c
function "static int winWrite")
    this was the output before the fail (ERROR_DISK_FULL)

...
WRITE file=00000030, buffer=00EDBE08, amount=1024, offset=4294965248, lock=4
OVERLAPPED offset=4294965248, OffsetHigh=0
WRITE file=00000030, buffer=00EDC2E0, amount=1024, offset=4294966272, lock=4
OVERLAPPED offset=4294966272, OffsetHigh=0

    Now note that

       4294966272 = 0xFFFFFC00
       and FFFFFFFF - FFFFFC00 = 3FF = 1023 !!!!! and we want to write
1024 bytes!!!


That this is a Microsoft bug is only a guess for what I have seen
debugging the code,
anyway if not I hope this analysis helps to find it the final cause of
the problem and it could be fixed.

Regards
Alejandro






_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

SQLITE_DISK_FULL_BUG.txt (3K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: BUG in 3.8.8.3 and pre-release: Wrong ERROR_DISK_FULL writing a blob on Windows

javaj1811@elxala.com
[hidden email] wrote:

> Hi, this is actually a Ticket, the good new is that is already
> pre-analyzed
> and probably also the cause has been found.
> I hope this is the right place to put this
>
> TITLE: Wrong ERROR_DISK_FULL writing a blob on Windows
>
> DESCRIPTION:
>
>    Having a sqlite database about 4.1 GB
>    writing blobs in some table, for instance using a statement like
>       INSERT INTO inodeCargo VALUES (1000, readfile ("ScreenShot.png"));
>
>    an error ERROR_DISK_FULL can be through wrongly (under certain
> conditions)
>
>    Windows 7 64-bit
>
> REPRODUCIBLE:
>    When the conditions are met, always reproducible
>
>
> CAUSE :
>
>    windows function WriteFile return error 112 (ERROR_DISK_FULL) when
> trying to write
>    1024 bytes using a offset = 4294966272 = 0xFFFFFC00 = FFFFFFFF - 2023
>
>    This was observed using the default mode (OVERLAPPED) although a
> test compiling
>    the code with
>       #define SQLITE_WIN32_NO_OVERLAPPED
>    at the begining also failed.
>
> PATCH:
>
>    The patch (see sqlite3_modified.c) work in my case, but obviously
> cannot
>    fix the mode NO_OVERLAPPED (it does not compile in this mode). Also
> all procedures
>    using the buggy function (if this result to be the cause) should be
> reviewed.
>    A more precise fix has to be developed by some expert in sqlite code.
>
> ANALYSIS:
>
>    A sqlite db used to store files came in the conditions described
>    size of database file : 4.194.303 KB
>    size of journal file : 8 KB
>    size of the image (png file) : 4.417 KB
>
>    SQL Query returning systematically ERROR_DISK_FULL:
>       INSERT INTO inodeCargo VALUES (-24, 4522876, 0, readfile
> ("ScreenShot320.png"));
>
>    (hard disk of the test with about 1 TB free space)
>
>    Since the case was 100% reproducible I could debug the problem using
>    sqlite-amalgamation-3080803.zip (ver 3.8.8.3)
>    I tried sqlite-amalgamation-201503091040.zip as well with the same
>    result, so the problem was also not fixed in this pre-release.
>
>    In debug session I found that the error came from
>
>       static int winWrite (...
>
>    and it is given in the line
>
>       if( !osWriteFile(pFile->h, aRem, nRem, &nWrite, &overlapped) ){
>
>    which is a call to a windows api function WriteFile
>
>    adding the extra debug messages into a file (see sqlite3_modified.c
> function "static int winWrite")
>    this was the output before the fail (ERROR_DISK_FULL)
>
> ...
> WRITE file=00000030, buffer=00EDBE08, amount=1024, offset=4294965248,
> lock=4
> OVERLAPPED offset=4294965248, OffsetHigh=0
> WRITE file=00000030, buffer=00EDC2E0, amount=1024, offset=4294966272,
> lock=4
> OVERLAPPED offset=4294966272, OffsetHigh=0
>
>    Now note that
>
>       4294966272 = 0xFFFFFC00
>       and FFFFFFFF - FFFFFC00 = 3FF = 1023 !!!!! and we want to write
> 1024 bytes!!!
>
>
> That this is a Microsoft bug is only a guess for what I have seen
> debugging the code,
> anyway if not I hope this analysis helps to find it the final cause of
> the problem and it could be fixed.
>
> Regards
> Alejandro
>
>
>
>
>
>
>
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
In the previous email (from another account of mine) I did attach a
zipped sqlite3_modified.c that
did not appear in the email published, maybe because of its size about
1MB although it was 7zipped
anyway, now I attach the code of only function modified in sqlite3.c
amalgamation: winWrite
here the relevant code where the patch lies


#if !SQLITE_OS_WINCE && !defined(SQLITE_WIN32_NO_OVERLAPPED)
     memset(&overlapped, 0, sizeof(OVERLAPPED));
     overlapped.Offset = (LONG)(offset & 0xffffffff);
     overlapped.OffsetHigh = (LONG)((offset>>32) & 0x7fffffff);

    // PATCH  !! ------------
    //
     if (((LONG)0xffffffff - (LONG)overlapped.Offset) < nRem)
       nRem = ((LONG)0xffffffff - (LONG)overlapped.Offset);
    // -----------


    // EXTRA DEBUG ------------
    //
    FILE * fout = fopen("salgo.txt", "at");
     if (fout != 0)
     {
         fprintf(fout, "OVERLAPPED offset=%u, OffsetHigh=%u\n",
overlapped.Offset, overlapped.OffsetHigh);
         fclose(fout);
     }
    // -----------

#endif


_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

sqlite3_winWrite_modified.c (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: BUG in 3.8.8.3 and pre-release: Wrong ERROR_DISK_FULL writing a blob on Windows

javaj1811@elxala.com
[hidden email] wrote:

> [hidden email] wrote:
>> Hi, this is actually a Ticket, the good new is that is already
>> pre-analyzed
>> and probably also the cause has been found.
>> I hope this is the right place to put this
>>
>> TITLE: Wrong ERROR_DISK_FULL writing a blob on Windows
>>
>> DESCRIPTION:
>>
>>    Having a sqlite database about 4.1 GB
>>    writing blobs in some table, for instance using a statement like
>>       INSERT INTO inodeCargo VALUES (1000, readfile ("ScreenShot.png"));
>>
>>    an error ERROR_DISK_FULL can be through wrongly (under certain
>> conditions)
>>
>>    Windows 7 64-bit
>>
>> REPRODUCIBLE:
>>    When the conditions are met, always reproducible
>>
>>
>> CAUSE :
>>
>>    windows function WriteFile return error 112 (ERROR_DISK_FULL) when
>> trying to write
>>    1024 bytes using a offset = 4294966272 = 0xFFFFFC00 = FFFFFFFF - 2023
>>
>>    This was observed using the default mode (OVERLAPPED) although a
>> test compiling
>>    the code with
>>       #define SQLITE_WIN32_NO_OVERLAPPED
>>    at the begining also failed.
>>
>> PATCH:
>>
>>    The patch (see sqlite3_modified.c) work in my case, but obviously
>> cannot
>>    fix the mode NO_OVERLAPPED (it does not compile in this mode).
>> Also all procedures
>>    using the buggy function (if this result to be the cause) should
>> be reviewed.
>>    A more precise fix has to be developed by some expert in sqlite code.
>>
>> ANALYSIS:
>>
>>    A sqlite db used to store files came in the conditions described
>>    size of database file : 4.194.303 KB
>>    size of journal file : 8 KB
>>    size of the image (png file) : 4.417 KB
>>
>>    SQL Query returning systematically ERROR_DISK_FULL:
>>       INSERT INTO inodeCargo VALUES (-24, 4522876, 0, readfile
>> ("ScreenShot320.png"));
>>
>>    (hard disk of the test with about 1 TB free space)
>>
>>    Since the case was 100% reproducible I could debug the problem using
>>    sqlite-amalgamation-3080803.zip (ver 3.8.8.3)
>>    I tried sqlite-amalgamation-201503091040.zip as well with the same
>>    result, so the problem was also not fixed in this pre-release.
>>
>>    In debug session I found that the error came from
>>
>>       static int winWrite (...
>>
>>    and it is given in the line
>>
>>       if( !osWriteFile(pFile->h, aRem, nRem, &nWrite, &overlapped) ){
>>
>>    which is a call to a windows api function WriteFile
>>
>>    adding the extra debug messages into a file (see
>> sqlite3_modified.c function "static int winWrite")
>>    this was the output before the fail (ERROR_DISK_FULL)
>>
>> ...
>> WRITE file=00000030, buffer=00EDBE08, amount=1024, offset=4294965248,
>> lock=4
>> OVERLAPPED offset=4294965248, OffsetHigh=0
>> WRITE file=00000030, buffer=00EDC2E0, amount=1024, offset=4294966272,
>> lock=4
>> OVERLAPPED offset=4294966272, OffsetHigh=0
>>
>>    Now note that
>>
>>       4294966272 = 0xFFFFFC00
>>       and FFFFFFFF - FFFFFC00 = 3FF = 1023 !!!!! and we want to write
>> 1024 bytes!!!
>>
>>
>> That this is a Microsoft bug is only a guess for what I have seen
>> debugging the code,
>> anyway if not I hope this analysis helps to find it the final cause
>> of the problem and it could be fixed.
>>
>> Regards
>> Alejandro
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> sqlite-users mailing list
>> [hidden email]
>> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
> In the previous email (from another account of mine) I did attach a
> zipped sqlite3_modified.c that
> did not appear in the email published, maybe because of its size about
> 1MB although it was 7zipped
> anyway, now I attach the code of only function modified in sqlite3.c
> amalgamation: winWrite
> here the relevant code where the patch lies
>
>
> #if !SQLITE_OS_WINCE && !defined(SQLITE_WIN32_NO_OVERLAPPED)
>     memset(&overlapped, 0, sizeof(OVERLAPPED));
>     overlapped.Offset = (LONG)(offset & 0xffffffff);
>     overlapped.OffsetHigh = (LONG)((offset>>32) & 0x7fffffff);
>
>    // PATCH  !! ------------
>    //
>     if (((LONG)0xffffffff - (LONG)overlapped.Offset) < nRem)
>       nRem = ((LONG)0xffffffff - (LONG)overlapped.Offset);
>    // -----------
>
>
>    // EXTRA DEBUG ------------
>    //
>    FILE * fout = fopen("salgo.txt", "at");
>     if (fout != 0)
>     {
>         fprintf(fout, "OVERLAPPED offset=%u, OffsetHigh=%u\n",
> overlapped.Offset, overlapped.OffsetHigh);
>         fclose(fout);
>     }
>    // -----------
>
> #endif
>
>
>
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
also specify that the exact error given by sqlite in this problem is

#define SQLITE_FULL        13   /* Insertion failed because database is
full */
     /* SQLITE_FULL        */ *"database or disk is full",*

which is produced by the windows api error ERROR_DISK_FULL

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: BUG in 3.8.8.3 and pre-release: Wrong ERROR_DISK_FULL writing a blob on Windows

Joe Mistachkin-3

What file system is being used on the drive in question (e.g. FAT, FAT32,
NTFS, etc)?

--
Joe Mistachkin

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: BUG in 3.8.8.3 and pre-release: Wrong ERROR_DISK_FULL writing a blob on Windows

javaj1811@elxala.com
Joe Mistachkin wrote:

> What file system is being used on the drive in question (e.g. FAT, FAT32,
> NTFS, etc)?
>
> --
> Joe Mistachkin
>
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
FAT32 in the documented case, I can test it in a NTFS drive as well, I
need just the time to copy the database
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: BUG in 3.8.8.3 and pre-release: Wrong ERROR_DISK_FULL writing a blob on Windows

javaj1811@elxala.com
[hidden email] wrote:

> Joe Mistachkin wrote:
>> What file system is being used on the drive in question (e.g. FAT,
>> FAT32,
>> NTFS, etc)?
>>
>> --
>> Joe Mistachkin
>>
>> _______________________________________________
>> sqlite-users mailing list
>> [hidden email]
>> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>>
> FAT32 in the documented case, I can test it in a NTFS drive as well, I
> need just the time to copy the database
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
Hi Joe,

good point, I've tested the fail scenario having the database located in
a NTFS disk with the result of NO FAIL!!!
so the insertion was done without sqlite error.
now the problem seems to be reduced to Windows and FAT32 (and maybe FAT)
drives
could you tell me why did you suspect about the drive type ?




_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: BUG in 3.8.8.3 and pre-release: Wrong ERROR_DISK_FULL writing a blob on Windows

Joe Mistachkin-3

[hidden email] wrote:
>
> good point, I've tested the fail scenario having the database located in
> a NTFS disk with the result of NO FAIL!!!
> so the insertion was done without sqlite error.
> now the problem seems to be reduced to Windows and FAT32 (and maybe FAT)
> drives could you tell me why did you suspect about the drive type ?
>

The FAT and FAT32 file systems have a relatively low limit on the maximum
size of a single file.  Their limit happens to be right around 4GB.

--
Joe Mistachkin

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: BUG in 3.8.8.3 and pre-release: Wrong ERROR_DISK_FULL writing a blob on Windows

R Smith
In reply to this post by javaj1811@elxala.com


On 2015-03-16 12:49 AM, [hidden email] wrote:
> Hi Joe,
>
> good point, I've tested the fail scenario having the database located
> in a NTFS disk with the result of NO FAIL!!!
> so the insertion was done without sqlite error.
> now the problem seems to be reduced to Windows and FAT32 (and maybe
> FAT) drives
> could you tell me why did you suspect about the drive type ?

You do know what the "32" in FAT32 means right?  For all intents and
purposes, in a FAT32 system, that disk might actually be full. Maximum
size for a FAT32 disk used to be 32GB but there are some programs
available which can write a FAT32 allocation table with 512 byte sectors
up to about ~2TB - which I'm hoping is the case for you - but more
likely you are hitting the upper bound at 32GB. Even if you did manage
to extend the partition beyond 32GB, the maximum file size will be 1
byte shy of 4GB. If you exceed any of these hard limits, you will get a
disk-full error (correctly because in a FAT32 universe, that disk or
file is in fact "Full").

Btw, FAT32 must be the near worst file system ever used in history (and
I'm not blaming Microsoft, they had to try to come up with "something"
that could be backward compatible with FAT16 and provide larger storage,
but in normal use it is atrocious).  Is there a reason you need to use it?


_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: BUG in 3.8.8.3 and pre-release: Wrong ERROR_DISK_FULL writing a blob on Windows

javaj1811@elxala.com
R.Smith wrote:

>
>
> On 2015-03-16 12:49 AM, [hidden email] wrote:
>> Hi Joe,
>>
>> good point, I've tested the fail scenario having the database located
>> in a NTFS disk with the result of NO FAIL!!!
>> so the insertion was done without sqlite error.
>> now the problem seems to be reduced to Windows and FAT32 (and maybe
>> FAT) drives
>> could you tell me why did you suspect about the drive type ?
>
> You do know what the "32" in FAT32 means right?  For all intents and
> purposes, in a FAT32 system, that disk might actually be full. Maximum
> size for a FAT32 disk used to be 32GB but there are some programs
> available which can write a FAT32 allocation table with 512 byte
> sectors up to about ~2TB - which I'm hoping is the case for you - but
> more likely you are hitting the upper bound at 32GB. Even if you did
> manage to extend the partition beyond 32GB, the maximum file size will
> be 1 byte shy of 4GB. If you exceed any of these hard limits, you will
> get a disk-full error (correctly because in a FAT32 universe, that
> disk or file is in fact "Full").
>
> Btw, FAT32 must be the near worst file system ever used in history
> (and I'm not blaming Microsoft, they had to try to come up with
> "something" that could be backward compatible with FAT16 and provide
> larger storage, but in normal use it is atrocious).  Is there a reason
> you need to use it?
>
>
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>

Ok, the limit of 4GB for a file in FAT32 seems to be the cause of the
problem, I didn't realize it since until now
I never needed such big files in my home computer and also get confused
by the error message (definitively not disk full but file full).

answering your question
 >>> Is there a reason you need to use it?
The disk was already formatted with FAT32 when I bought it some years
ago. And indeed is the most reliable
external drive disk that I have in the last years but now I've learned
that is not appropiate for my big sqlite databases!

thank you for your feedback!









_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users