Cannot initialize statically linked extension

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Cannot initialize statically linked extension

Lifepillar
Hello,
I need some help to figure out a segfault. I have:

- SQLite3 3.21.0 compiled as a static library;
- a custom extension compiled as another static library,
   passing -DSQLITE_CORE.
- a simple main program linked to both libraries.

My extension has the typical structure, nothing fancy.

My main() function contains the following snippet:

  int rc = sqlite3_open(":memory:", &db);
   if (rc != SQLITE_OK) {
     sqlite3_close(db);
     return rc;
   }
   char* zErrMsg;
   //sqlite3_myextension_init(db, &zErrMsg, 0);

This works fine: I can use SQLite functions without problems.
But, if I un-comment the line that invokes the entry point of
my extension (the last line of the snippet above), the program
crashes with a bad memory access error.

Btw, if I build my extension as a dynamic library and load it
in a SQLite shell, all is fine.

Any idea what may be wrong?

Thanks,
Life.




_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Cannot initialize statically linked extension

Keith Medcalf

You should only be defining SQLITE_CORE if in fact the extension is part of the core -- that is compiled and included (statically linked) to the core sqlite3.c compilation unit.  In this case, the extension makes direct calls to the sqlite3 entry points and shares the same runtime as the sqlite3 core engine.

If the extension is *not* part of the same linkage unit as sqlite3.c, then it cannot access the sqlite3 core functions using direct linkage and must instead use a structure of function pointers (aka a class) for indirect access.

Having an extension be a part of the core but not telling it so is only deleterious to performance (linkage is via indirect function pointers rather than static linkage).

Telling an extension that it is part of the core when it is not will lead to memory access errors, either due to bad loadtime linkage or more likely to the fact that it is using a separate runtime heap and/or stack manager and neither the extension nor the actual sqlite3 core can manage each others' memory allocations.

---
The fact that there's a Highway to Hell but only a Stairway to Heaven says a lot about anticipated traffic volume.


>-----Original Message-----
>From: sqlite-users [mailto:sqlite-users-
>[hidden email]] On Behalf Of Lifepillar
>Sent: Monday, 4 December, 2017 12:45
>To: [hidden email]
>Subject: [sqlite] Cannot initialize statically linked extension
>
>Hello,
>I need some help to figure out a segfault. I have:
>
>- SQLite3 3.21.0 compiled as a static library;
>- a custom extension compiled as another static library,
>   passing -DSQLITE_CORE.
>- a simple main program linked to both libraries.
>
>My extension has the typical structure, nothing fancy.
>
>My main() function contains the following snippet:
>
>  int rc = sqlite3_open(":memory:", &db);
>   if (rc != SQLITE_OK) {
>     sqlite3_close(db);
>     return rc;
>   }
>   char* zErrMsg;
>   //sqlite3_myextension_init(db, &zErrMsg, 0);
>
>This works fine: I can use SQLite functions without problems.
>But, if I un-comment the line that invokes the entry point of
>my extension (the last line of the snippet above), the program
>crashes with a bad memory access error.
>
>Btw, if I build my extension as a dynamic library and load it
>in a SQLite shell, all is fine.
>
>Any idea what may be wrong?
>
>Thanks,
>Life.
>
>
>
>
>_______________________________________________
>sqlite-users mailing list
>[hidden email]
>http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users



_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Cannot initialize statically linked extension

Lifepillar
On 04/12/2017 20:59, Keith Medcalf wrote:
>
> You should only be defining SQLITE_CORE if in fact the extension is part of the core -- that is compiled and included (statically linked) to the core sqlite3.c compilation unit.  In this case, the extension makes direct calls to the sqlite3 entry points and shares the same runtime as the sqlite3 core engine.
>
> If the extension is *not* part of the same linkage unit as sqlite3.c, then it cannot access the sqlite3 core functions using direct linkage and must instead use a structure of function pointers (aka a class) for indirect access.
>
> Having an extension be a part of the core but not telling it so is only deleterious to performance (linkage is via indirect function pointers rather than static linkage).
>
> Telling an extension that it is part of the core when it is not will lead to memory access errors, either due to bad loadtime linkage or more likely to the fact that it is using a separate runtime heap and/or stack manager and neither the extension nor the actual sqlite3 core can manage each others' memory allocations.
Got it. I had misinterpreted the documentation in a way that—
now I realize—doesn't make much sense.

Thanks for the clear explanation!
Life.

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Cannot initialize statically linked extension

Jens Alfke-2
In reply to this post by Keith Medcalf


> On Dec 4, 2017, at 11:59 AM, Keith Medcalf <[hidden email]> wrote:
>
> You should only be defining SQLITE_CORE if in fact the extension is part of the core -- that is compiled and included (statically linked) to the core sqlite3.c compilation unit.

Which it is, in this case. The OP said that both sqlite and the extension are static libraries, so they’re both being linked directly into the executable.

I’m not sure what’s going on. Life, can you post a backtrace of the crash?

—Jens
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Cannot initialize statically linked extension

Keith Medcalf
On Monday, 4 December, 2017 15:03, Jens Alfke <[hidden email]> wrote:
>> On Dec 4, 2017, at 11:59 AM, Keith Medcalf <[hidden email]>
>>wrote:

>> You should only be defining SQLITE_CORE if in fact the extension is
>>part of the core -- that is compiled and included (statically linked)
>>to the core sqlite3.c compilation unit.

>Which it is, in this case. The OP said that both sqlite and the
>extension are static libraries, so they’re both being linked directly
>into the executable.

Hmmm.  You are right.  I read is as statically linked loadlibs (DLLs) not as all the object modules were linked together into a single executable; then you are right, there should not be an issue.  However, each compilation unit still must be using the same single uniform runtime library.  If one object is using, for example, the multithreaded runtime and the others are using the single threaded runtime (for example), and the third perhaps the subsystem runtime, you will end up with the same runtime collisions after they are linked together.

>I’m not sure what’s going on. Life, can you post a backtrace of the
>crash?

And perhaps use the dependency walker (depends.exe) on the executable and see if multiple runtimes are being loaded ...




_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Cannot initialize statically linked extension

Jens Alfke-2


> On Dec 4, 2017, at 2:25 PM, Keith Medcalf <[hidden email]> wrote:
>
> If one object is using, for example, the multithreaded runtime and the others are using the single threaded runtime (for example), and the third perhaps the subsystem runtime

From the OP’s other thread here it looks like they’re developing for iOS or macOS, neither of which have the sorts of multiple runtimes you’re talking about (is that a Windows thing?)

Also, I don’t see how static libraries can end up using different runtime libraries, since the runtime gets linked in by the linker, after the static libs are compiled. But maybe that’s another Windows thing.

—Jens
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Cannot initialize statically linked extension

Keith Medcalf
On Monday, 4 December, 2017 15:44, Jens Alfke <[hidden email]> wrote:

>> If one object is using, for example, the multithreaded runtime and
>>the others are using the single threaded runtime (for example), and
>>the third perhaps the subsystem runtime

>From the OP’s other thread here it looks like they’re developing for
>iOS or macOS, neither of which have the sorts of multiple runtimes
>you’re talking about (is that a Windows thing?)

I had believed it was Windows since "DLL" was mentioned and that is a Windows (or OS/2) contraction for dynamic load library.  I don't think anything else uses that term in quite the same way.

The multiple library is a Microsoft thing designed to ensure a continuous revenue stream by ensuring that each version of their development platform is incompatible with all prior versions.  Once you start using Microsoft tools for anything you are stuck (locked in) into a perpetual cycle of re-writing and re-development of every single thing due to their planned obsolescence requiring re-purchasing tools and redeveloping everything practically fortnightly.  The "newer fangled" the technology (from a Microsoft hype perspective) the shorter its viability.  This is in contrast to where, for example, one avoids Microsoft tools like the plague, and you end up with stuff that works perfectly for decades on multiple versions of Windows.  This does not generate continuous upgrade revenue for either Microsoft or the developer (if it ain't broke why fix it) so typically there is a preponderance of deliberate lock-in by all parties in the Microsoft world.

>Also, I don’t see how static libraries can end up using different
>runtime libraries, since the runtime gets linked in by the linker,
>after the static libs are compiled. But maybe that’s another Windows
>thing.

Because for the greater certainty of lock-in, the runtime entry point names are different depending on the version of the Toolchain and the particular link library you compiled the object for.  In many cases you cannot simply remove the linkage records from the objects and link with a substitute library (though sometimes you can).  You have to rebuild the whole thing again. You can link against the platform subsystem library, but that is recommended against because it makes the resultant executable too portable across versions of Windows.





_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Cannot initialize statically linked extension

Guy Harris
On Dec 4, 2017, at 3:42 PM, Keith Medcalf <[hidden email]> wrote:

> On Monday, 4 December, 2017 15:44, Jens Alfke <[hidden email]> wrote:
>
>>> If one object is using, for example, the multithreaded runtime and
>>> the others are using the single threaded runtime (for example), and
>>> the third perhaps the subsystem runtime
>
>> From the OP’s other thread here it looks like they’re developing for
>> iOS or macOS, neither of which have the sorts of multiple runtimes
>> you’re talking about (is that a Windows thing?)
>
> I had believed it was Windows since "DLL" was mentioned

No, the original message said

> Btw, if I build my extension as a dynamic library and load it
> in a SQLite shell, all is fine.

which says "dynamic library", rather than "DLL"; the term "dynamic" is used with most UN*X implementations of shared libraries (and the file suffix used for shared libraries on Darwin-based systems such as macOS and iOS is ".dylib").

> and that is a Windows (or OS/2) contraction for dynamic load library.  I don't think anything else uses that term in quite the same way.

The initialism "DLL" is a sign that the system being discussed is almost certainly Windows.  The word "dynamic", however, isn't - it may even be a sign that it's a UN*X, as they probably would have said "DLL" were it Windows.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Cannot initialize statically linked extension

Lifepillar
On 05/12/2017 09:24, Guy Harris wrote:
 > On Dec 4, 2017, at 3:42 PM, Keith Medcalf <[hidden email]> wrote:
 >
 >> On Monday, 4 December, 2017 15:44, Jens Alfke <[hidden email]>
wrote:
 >>
 >>>> If one object is using, for example, the multithreaded runtime and
 >>>> the others are using the single threaded runtime (for example), and
 >>>> the third perhaps the subsystem runtime
 >>
 >>>  From the OP’s other thread here it looks like they’re developing
 >>>  for iOS or macOS, neither of which have the sorts of multiple
 >>>  runtimes you’re talking about (is that a Windows thing?)
 >>
 >> I had believed it was Windows since "DLL" was mentioned
 >
 > No, the original message said
 >
 >> Btw, if I build my extension as a dynamic library and load it
 >> in a SQLite shell, all is fine.
 >
 > which says "dynamic library", rather than "DLL"; the term "dynamic" is
 > used with most UN*X implementations of shared libraries (and the file
 > suffix used for shared libraries on Darwin-based systems such as macOS
 > and iOS is ".dylib").
 >
 >> and that is a Windows (or OS/2) contraction for dynamic load library.
 >> I don't think anything else uses that term in quite the same way.
 >
 > The initialism "DLL" is a sign that the system being discussed is
 > almost certainly Windows.  The word "dynamic", however, isn't - it may
 > even be a sign that it's a UN*X, as they probably would have said
 > "DLL" were it Windows.

My fault, I should have mentioned my platform's details. As already
pointed out, I'm on macOS, and, specifically, I'm building my project
with Xcode.

On Dec 4, 2017, at 11:59 AM, Keith Medcalf <[hidden email]> wrote:
 >>
 >> You should only be defining SQLITE_CORE if in fact the extension is
 >> part of the core -- that is compiled and included (statically linked)
 >> to the core sqlite3.c compilation unit.
 >
 > Which it is, in this case. The OP said that both sqlite and the
 > extension are static libraries, so they’re both being linked directly
 > into the executable.
 >
 > I’m not sure what’s going on. Life, can you post a backtrace of the
 > crash?

I'll try to make a minimal reproducible example. For now, I have fixed
my issue by linking my extension to the sqlite library rather than to my
executable.

Life.


_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Cannot initialize statically linked extension

Lifepillar
>  > Which it is, in this case. The OP said that both sqlite and th
>  > extension are static libraries, so they’re both being linked directly
>  > into the executable.
>  >
>  > I’m not sure what’s going on. Life, can you post a backtrace of the
>  > crash?
>
> I'll try to make a minimal reproducible example. For now, I have fixed
> my issue by linking my extension to the sqlite library rather than to my
> executable.

So, it is most likely a bug in Xcode :/

Probably most of you don't care, but I will describe it just for future
reference.

The issue is related to the fact that I have two targets to build my
extension as a static library and as a dynamic library, respectively.

For some reason that goes beyond my understanding, if the .dylib exists,
my main target picks it up instead of the static library, although I
have explicitly declared only the static library as a dependency, and
even if "Find implicit dependencies" is unchecked. If I remove the
.dylib file, then the main target correctly links to the static library.

Interestingly, I have this issue only with the Debug configuration: if I
build in Release mode, everything builds and runs as expected. Even more
interestingly, if I build in Debug mode, but from the command line,
everything builds and runs as expected (even when the .dylib is there).

What can I say? After the "login as root without password" bug, I should
expect anything... Anyway, your comments helped me narrowing down the
problem.

Sorry for the noise guys!
Life.

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Cannot initialize statically linked extension

Steve Harrill
In reply to this post by Lifepillar
Get me off of this list

Sent from my iPhone

On Dec 5, 2017, at 04:48, Lifepillar <[hidden email]> wrote:

On 05/12/2017 09:24, Guy Harris wrote:

> On Dec 4, 2017, at 3:42 PM, Keith Medcalf <[hidden email]> wrote:
>
>> On Monday, 4 December, 2017 15:44, Jens Alfke <[hidden email]> wrote:
>>
>>>> If one object is using, for example, the multithreaded runtime and
>>>> the others are using the single threaded runtime (for example), and
>>>> the third perhaps the subsystem runtime
>>
>>>  From the OP’s other thread here it looks like they’re developing
>>>  for iOS or macOS, neither of which have the sorts of multiple
>>>  runtimes you’re talking about (is that a Windows thing?)
>>
>> I had believed it was Windows since "DLL" was mentioned
>
> No, the original message said
>
>> Btw, if I build my extension as a dynamic library and load it
>> in a SQLite shell, all is fine.
>
> which says "dynamic library", rather than "DLL"; the term "dynamic" is
> used with most UN*X implementations of shared libraries (and the file
> suffix used for shared libraries on Darwin-based systems such as macOS
> and iOS is ".dylib").
>
>> and that is a Windows (or OS/2) contraction for dynamic load library.
>> I don't think anything else uses that term in quite the same way.
>
> The initialism "DLL" is a sign that the system being discussed is
> almost certainly Windows.  The word "dynamic", however, isn't - it may
> even be a sign that it's a UN*X, as they probably would have said
> "DLL" were it Windows.

My fault, I should have mentioned my platform's details. As already
pointed out, I'm on macOS, and, specifically, I'm building my project
with Xcode.

On Dec 4, 2017, at 11:59 AM, Keith Medcalf <[hidden email]> wrote:

>>
>> You should only be defining SQLITE_CORE if in fact the extension is
>> part of the core -- that is compiled and included (statically linked)
>> to the core sqlite3.c compilation unit.
>
> Which it is, in this case. The OP said that both sqlite and the
> extension are static libraries, so they’re both being linked directly
> into the executable.
>
> I’m not sure what’s going on. Life, can you post a backtrace of the
> crash?

I'll try to make a minimal reproducible example. For now, I have fixed
my issue by linking my extension to the sqlite library rather than to my
executable.

Life.


_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users