Cross-compiling fails.

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Cross-compiling fails.

Alastair Growcott
Cross-compiling sqlite3 fails due to the use of binaries in the build
process that are generated during the build process. Specifically:

     mksourceid
     lemon

The configure script explicitly offers the "--host=" option to allow
cross-compiling. If cross-compiling is not supported then this option
should be removed, and ideally replace with a short paragraph of text
explaining this explicitly.

An alternative is to add steps to the build instructions explaining that
you need to build these binaries natively first and then put them in the
correct location - however they will probably still get over-written by
cross-compiled versions when the cross-compile build starts so there may
need to be additional changes made.

In addition, when cross-compiling for Windows with the MinGW toolchain,
the code looks for the file "sys/ioctl.h" which doesn't exist for
Windows/MinGW systems. It looks like SQLITE_OS_UNIX is being incorrectly
defined. It doesn't look like a difficult problem to solve and if I get
a reply I will be happy to share any solution I find.


Alastair.

--
Senior Software Engineer
H Scientific Ltd
023 9226 7607
www.h-scientific.co.uk


---
This email has been checked for viruses by AVG.
http://www.avg.com

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Cross-compiling fails.

Richard Hipp-3
On 12/4/17, Alastair Growcott <[hidden email]> wrote:
> Cross-compiling sqlite3 fails due to the use of binaries in the build
> process that are generated during the build process.

I recommend that you do

    ./configure; make sqlite3.c

or

    nmake /f makefile.msc sqlite3.c

depending on whether your host system is unix or windows.  Then
manually compile the resulting sqlite3.c for your target platform.

What are the host and target systems for the build you are trying to do?

--
D. Richard Hipp
[hidden email]
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Cross-compiling fails.

Joe Mistachkin-3
In reply to this post by Alastair Growcott

Alastair Growcott wrote:
>
> Cross-compiling sqlite3 fails due to the use of binaries in the
> build process that are generated during the build process.
> Specifically:
>

You'll need to use the NMAKE macro XCOMPILE.  Also, depending on
target platform you may need to use the other NMAKE macros like
USE_NATIVE_LIBPATHS, NCRTLIBPATH, and NSDKLIBPATH.

For example,
 
nmake /f Makefile.msc sqlite3.exe XCOMPILE=1 USE_NATIVE_LIBPATHS=1

Alternatively, you can build the amalgamation source code file
"sqlite3.c" without cross-compiling and then cross-compile it.

--
Joe Mistachkin @ https://urn.to/r/mistachkin

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Cross-compiling fails.

Alastair Growcott
In reply to this post by Richard Hipp-3
Thanks for the reply.

To answer your question using autoconf/configure terminology, my build
system is Linux and my host system is MinGW32. That is I am building on
Linux to generate libraries/executables to run under MinGW (on Windows
essentially). I would however face exactly the same issue if I was
building on Linux to run on, say, a RaspberryPi or an Android system.

To avoid confusion I recommend that you remove the option to
cross-compile unless you seriously intend to support it in which case
you will need to make some serious changes to your configure.ac script.
In any case you should remove the "--target" option as that is only used
when building compilers (I am building on system A (build) a compiler
which will run on system B (host) which will generate binaries that run
on system C (target)). The determination of the executable extension is
also broken with some confusion over "build" system extension and
"target" system extension. The script is incorrectly determining that
the "build" system extension is ".exe" by building an executable using
the cross-compiler. It then incorrectly determines that the target
extension is non-existent because it seems to somehow be assigning the
build executable extension to that variable. It then goes on to decide
that because of this the target OS is *nix and then SQLite fails to
compile because some platform compatibility code tries to include a *nix
specific header file. I solved this by simply deleting chunks of code in
the configure script and setting the target extension to be the same as
the build extension. However this is a hack and not a proper solution.

Also, I am not sure your suggestion of generating sqlite3.c natively and
then cross-compiling it separately will work. The generated sqlite3.c
has some platform dependent stuff (selective header inclusion as per
above) in it.


Senior Software Engineer
H Scientific Ltd
023 9226 7607
www.h-scientific.co.uk

On 06/12/2017 02:10, Richard Hipp wrote:

> On 12/4/17, Alastair Growcott <[hidden email]> wrote:
>> Cross-compiling sqlite3 fails due to the use of binaries in the build
>> process that are generated during the build process.
> I recommend that you do
>
>      ./configure; make sqlite3.c
>
> or
>
>      nmake /f makefile.msc sqlite3.c
>
> depending on whether your host system is unix or windows.  Then
> manually compile the resulting sqlite3.c for your target platform.
>
> What are the host and target systems for the build you are trying to do?
>


---
This email has been checked for viruses by AVG.
http://www.avg.com

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Cross-compiling fails.

Richard Hipp-3
On 12/6/17, Alastair Growcott <[hidden email]> wrote:
>
> To avoid confusion I recommend that you remove the option to
> cross-compile

I'm cool with that approach.  Can you suggest a specific edit to the
configure.ac file that will accomplish what you describe?

>
> Also, I am not sure your suggestion of generating sqlite3.c natively and
> then cross-compiling it separately will work. The generated sqlite3.c
> has some platform dependent stuff (selective header inclusion as per
> above) in it.
>

The generated sqlite3.c file should be byte-for-byte identical on all
host platforms.  We test for that.  And we publish the SHA3 hash of
the generated sqlite3.c file in our release notes.  If you find a pair
of host platforms that build different "sqlite3.c" files from the same
sources, then that is a bug.  Please report it.

--
D. Richard Hipp
[hidden email]
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Cross-compiling fails.

Alastair Growcott
Trying to get it to work correctly:

BUILD_CC is correctly determined, but BUILD_EXEEXT is not (unless you
have a Cygwin system). On closer examination, the system is in fact
natively building lemon and mksourceid but because the extension is
wrong, you get an error like:

    make: *** No rule to make target 'lemon', needed by 'fts5parse.c'. Stop.

Simply calculating BUILD_EXEEXT correctly should make this bit work, and
cross-compilation would be nice to have. I have now spent way way too
long, while being paid to do other stuff, looking at this. Sorry I can't
do more.

Also TARGET_EXEEXT is calculated correctly. In fact because you are not
building a compiler this value is not needed - simple substitute EXEEXT
everywhere you use TARGET_EXEEXT. E.g. change:

    if test x"$cross_compiling" = xno; then
       TARGET_EXEEXT=$BUILD_EXEEXT
    else
       TARGET_EXEEXT=$config_TARGET_EXEEXT
    fi
    if test "$TARGET_EXEEXT" = ".exe"; then
       SQLITE_OS_UNIX=0
       SQLITE_OS_WIN=1
       CFLAGS="$CFLAGS -DSQLITE_OS_WIN=1"
    else
       SQLITE_OS_UNIX=1
       SQLITE_OS_WIN=0
       CFLAGS="$CFLAGS -DSQLITE_OS_UNIX=1"
    fi

to:

    if test "$EXEEXT" = ".exe"; then
       SQLITE_OS_UNIX=0
       SQLITE_OS_WIN=1
       CFLAGS="$CFLAGS -DSQLITE_OS_WIN=1"
    else
       SQLITE_OS_UNIX=1
       SQLITE_OS_WIN=0
       CFLAGS="$CFLAGS -DSQLITE_OS_UNIX=1"
    fi

And in Makefile.in change @TARGET_EXEEXT@ to @EXEEXT@. These changes are
untested (no time).



Trying to remove cross-compilation:

At the top of configure.ac is the following text that I recommend you
edit for starters:

# The build process allows for using a cross-compiler.  But the default
# action is to target the same platform that we are running on.  The
# configure script needs to discover the following properties of the
# build and target systems:


Search the configure.ac for "cross_compiling". Change everything that
needs changing, or remove it. I notice that there is code there to
distinguish the build executable extension from the host one, but it
doesn't work. I spent a while that I should have spent on paid work
trying to figure out how to get that to work and in conclusion, it doesn't.

As far as the autoconf stuff goes, from the configure options it looks
like you have AC_CANONICAL_HOST used, but I only see it in aclocal.m4. I
don't know enough about autoconf to help here - I don't really know what
aclocal.m4 is used for, what it does.


Having re-examined sqlite3.c I think you are right that we can compile
it separately. I think the information needs to be put in a README
somewhere. It will need instructions on which defines and flags need to
be set. Normally configure sets that all up for you.



The way I built it in the end was as follows:

    mkdir sqlite_linux
    cd sqlite_linux
    ../sqlite/configure
    Interrupt the build early, once lemon and mksourceid have been built
    cd ..
    mkdir sqlite_mingw32
    cd sqlite_mingw32
    Edit ../sqlite/configure around line 270 to replace:

        if test x"$cross_compiling" = xno; then
           TARGET_EXEEXT=$BUILD_EXEEXT
        else
           TARGET_EXEEXT=$config_TARGET_EXEEXT
        fi

    with something like:

        TARGET_EXEEXT=$EXEEXT

    $config_TARGET_EXEEXT is not set.

    ../sqlite/configure --host=i686-w64-mingw32
    cp ../sqlite_linux/lemon .
    cp ../sqlite_linux/mksourceid .
    make

However it does look like all I really needed to do was to create a
symlink or something from lemon to lemon.exe and the same for
mksourceid, plus of course fixing the calculation of TARGET_EXEEXT which
just needs to be the same as EXEEXT, or even better just use EXEEXT and
not TARGET_EXEEXT.


Senior Software Engineer
H Scientific Ltd
023 9226 7607
www.h-scientific.co.uk

On 06/12/2017 08:24, Richard Hipp wrote:

> On 12/6/17, Alastair Growcott <[hidden email]> wrote:
>> To avoid confusion I recommend that you remove the option to
>> cross-compile
> I'm cool with that approach.  Can you suggest a specific edit to the
> configure.ac file that will accomplish what you describe?
>
>> Also, I am not sure your suggestion of generating sqlite3.c natively and
>> then cross-compiling it separately will work. The generated sqlite3.c
>> has some platform dependent stuff (selective header inclusion as per
>> above) in it.
>>
> The generated sqlite3.c file should be byte-for-byte identical on all
> host platforms.  We test for that.  And we publish the SHA3 hash of
> the generated sqlite3.c file in our release notes.  If you find a pair
> of host platforms that build different "sqlite3.c" files from the same
> sources, then that is a bug.  Please report it.
>


---
This email has been checked for viruses by AVG.
http://www.avg.com

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Cross-compiling fails.

Alastair Growcott
In reply to this post by Richard Hipp-3
I just can't let some things go. I am such a bad boy.

See attachment. It has a great big TODO when working out BUILD_EXEEXT
that still needs implementing. Note that you need to change Makefile.in
as well to change "@TARGET_EXEEXT@" to "@EXEEXT@".

Tested as working on my system and configuration. Needs additional
testing once properly implemented the TODO.


Senior Software Engineer
H Scientific Ltd
023 9226 7607
www.h-scientific.co.uk

On 06/12/2017 08:24, Richard Hipp wrote:

> On 12/6/17, Alastair Growcott <[hidden email]> wrote:
>> To avoid confusion I recommend that you remove the option to
>> cross-compile
> I'm cool with that approach.  Can you suggest a specific edit to the
> configure.ac file that will accomplish what you describe?
>
>> Also, I am not sure your suggestion of generating sqlite3.c natively and
>> then cross-compiling it separately will work. The generated sqlite3.c
>> has some platform dependent stuff (selective header inclusion as per
>> above) in it.
>>
> The generated sqlite3.c file should be byte-for-byte identical on all
> host platforms.  We test for that.  And we publish the SHA3 hash of
> the generated sqlite3.c file in our release notes.  If you find a pair
> of host platforms that build different "sqlite3.c" files from the same
> sources, then that is a bug.  Please report it.
>


---
This email has been checked for viruses by AVG.
http://www.avg.com

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

configure.ac (30K) Download Attachment