Problems with compiling SQLite for MUSL.

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Problems with compiling SQLite for MUSL.

John Found

I am trying to compile SQLite with 32bit MUSL, but probably mess some compiler options.

The command I am using is:

musl-gcc -O3 -shared -fno-stack-protector -DSQLITE_DEFAULT_MEMSTATUS=0 -DSQLITE_LIKE_DOESNT_MATCH_BLOBS -DSQLITE_MAX_EXPR_DEPTH=0 -DSQLITE_OMIT_DEPRECATED -D SQLITE_OMIT_PROGRESS_CALLBACK -DSQLITE_OMIT_SHARED_CACHE -DSQLITE_ENABLE_STAT4 -DSQLITE_ENABLE_FTS5 -DSQLITE_OMIT_COMPLETE -DSQLITE_OMIT_GET_TABLE -DSQLITE_OMIT_UTF16 -DSQLITE_THREADSAFE=1 ./sqlite3.c -o ./libsqlite3.so

The code compiles without errors. The problem is that the code is compiled with SSE instructions, but in runtime crashes on reads and writes to unaligned memory addresses. In most cases the crashes are random, but mostly in sqlite3_prepare_v2 function;

So, the question, how to make sqlite to compile with properly aligned variables in order to not crash. Is there some option about this (my gcc and generally C/C++ skills are pretty low) or I somehow messed the options elsewhere. Then what to check?


--
John Found <[hidden email]>
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Problems with compiling SQLite for MUSL.

Keith Medcalf

This is a compiler issue and has nothing to do with SQLite3.  Anything you compiled using that compiler would exhibit the same problems ...

The source for SQLite3 does not change the default compiler packing or alignment, so whatever alignment the compiler thinks is appropriate is what gets used.  

That said, the packing overrides for GCC (they are however dependent on the target CPU, and there are a myriad of different options which vary by target CPU and how the compiler was built) are likely:

-fpack-struct=n (where n is in bytes)
-mstructure-size-boundary=b (where b is in bits)
-malign-double (presumably this means double-word boundaries, or 64-bits)

However, that the compiler does not automatically use the correct alignment for the cpu architecture in use is puzzling, since this is something that one would normally expect to be correct by default.  You did not say what Operating System or CPU you are using ... this is probably important information.  As is likely the version of GCC (use gcc -v or musl-gcc -v to get the compiler's compile configuration).

---
The fact that there's a Highway to Hell but only a Stairway to Heaven says a lot about anticipated traffic volume.

>-----Original Message-----
>From: sqlite-users [mailto:sqlite-users-
>[hidden email]] On Behalf Of John Found
>Sent: Sunday, 2 September, 2018 12:18
>To: [hidden email]
>Subject: [sqlite] Problems with compiling SQLite for MUSL.
>
>
>I am trying to compile SQLite with 32bit MUSL, but probably mess some
>compiler options.
>
>The command I am using is:
>
>musl-gcc -O3 -shared -fno-stack-protector -
>DSQLITE_DEFAULT_MEMSTATUS=0 -DSQLITE_LIKE_DOESNT_MATCH_BLOBS -
>DSQLITE_MAX_EXPR_DEPTH=0 -DSQLITE_OMIT_DEPRECATED -D
>SQLITE_OMIT_PROGRESS_CALLBACK -DSQLITE_OMIT_SHARED_CACHE -
>DSQLITE_ENABLE_STAT4 -DSQLITE_ENABLE_FTS5 -DSQLITE_OMIT_COMPLETE -
>DSQLITE_OMIT_GET_TABLE -DSQLITE_OMIT_UTF16 -DSQLITE_THREADSAFE=1
>./sqlite3.c -o ./libsqlite3.so
>
>The code compiles without errors. The problem is that the code is
>compiled with SSE instructions, but in runtime crashes on reads and
>writes to unaligned memory addresses. In most cases the crashes are
>random, but mostly in sqlite3_prepare_v2 function;
>
>So, the question, how to make sqlite to compile with properly aligned
>variables in order to not crash. Is there some option about this (my
>gcc and generally C/C++ skills are pretty low) or I somehow messed
>the options elsewhere. Then what to check?
>
>
>--
>John Found <[hidden email]>
>_______________________________________________
>sqlite-users mailing list
>[hidden email]
>http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users



_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Problems with compiling SQLite for MUSL.

John Found
On Sun, 02 Sep 2018 14:18:50 -0600
"Keith Medcalf" <[hidden email]> wrote:

>
> This is a compiler issue and has nothing to do with SQLite3.  Anything you compiled using that compiler would exhibit the same problems ...

Well, I was almost sure, but as long as here I can find people understanding gcc and I am compiling sqlite after all,
considered a good place to ask. ;)

> However, that the compiler does not automatically use the correct alignment for the cpu architecture in use is puzzling, since this is something that one would normally expect to be correct by default.  You did not say what Operating System or CPU you are using ... this is probably important information.  As is likely the version of GCC (use gcc -v or musl-gcc -v to get the compiler's compile configuration).

The OS is 64bit Linux (Manjaro/Arch), the CPU is AMD A4-1200 (supports almost all extensions), gcc version is 8.2.0;

BTW, compiling with -O1 sometimes produces working result (but pretty slow) which only entagles the puzzle.

--
John Found <[hidden email]>
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Problems with compiling SQLite for MUSL.

Keith Medcalf

On Sunday, 2 September, 2018 20:32, John Found <[hidden email]> wrote:

>On Sun, 02 Sep 2018 14:18:50 -0600 "Keith Medcalf" <[hidden email]> wrote:

>> This is a compiler issue and has nothing to do with SQLite3.
>>Anything you compiled using that compiler would exhibit the same
>>problems ...

>Well, I was almost sure, but as long as here I can find people
>understanding gcc and I am compiling sqlite after all,
>considered a good place to ask. ;)

>> However, that the compiler does not automatically use the correct
>>alignment for the cpu architecture in use is puzzling, since this is
>>something that one would normally expect to be correct by default.
>>You did not say what Operating System or CPU you are using ... this
>>is probably important information.  As is likely the version of GCC
>>(use gcc -v or musl-gcc -v to get the compiler's compile
>>configuration).

>The OS is 64bit Linux (Manjaro/Arch), the CPU is AMD A4-1200
>(supports almost all extensions), gcc version is 8.2.0;

>BTW, compiling with -O1 sometimes produces working result (but pretty
>slow) which only entagles the puzzle.

Interesting ... Sounds like the optimizer in the compiler is broken ... unless someone has ideas about how to debug this.  Can you compile with no optimization and SQLITE_DEBUG defined and see what happens?  Though, to me it sounds like the compiler and/or the optimizer is just broken (or perhaps the replacement standard library is broken and does not return properly aligned memory allocations) ...

---
The fact that there's a Highway to Hell but only a Stairway to Heaven says a lot about anticipated traffic volume.



>
>--
>John Found <[hidden email]>
>_______________________________________________
>sqlite-users mailing list
>[hidden email]
>http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users



_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Problems with compiling SQLite for MUSL.

John Found
On Sun, 02 Sep 2018 22:25:16 -0600
"Keith Medcalf" <[hidden email]> wrote:

> Interesting ... Sounds like the optimizer in the compiler is broken ... unless someone has ideas about how to debug this.  Can you compile with no optimization and SQLITE_DEBUG defined and see what happens?  Though, to me it sounds like the compiler and/or the optimizer is just broken (or perhaps the replacement standard library is broken and does not return properly aligned memory allocations) ...

Well, the problem is solved. The crashes was caused because gcc compiled sqlite with requirement for stack alignment on 16 bytes,
regardles of that it was compiled to 32bit code. On the other hand, my code aligns the stack on 4 bytes as an usual 32bit program.

The option -mpreferred-stack-boundary=2 fixed the issue.

--
John Found <[hidden email]>
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users