cannot compile sqlite3 with intel compiler (ICC) - log and hints for a possible solution

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

cannot compile sqlite3 with intel compiler (ICC) - log and hints for a possible solution

Roberto C.
Hi,

while trying to install and sqlite package for R statistical language
(Rsqlite)  I came across I bug: i could not compile the package with the
intel compiler ICC / ICPC. You can find the whole discussion (and the
solution) here - https://github.com/rstats-db/RSQLite/issues/223

However, the maintainer suggested me to take the issue upstream to the
sqlite mailing list. Therefore, here I am.

The diagnosis of the problem is:

The undefined symbol, __builtin_mul_overflow, is a GCC primitive. SQLite
has the option to call it directly. It's supposed to disable itself through
a some pre-processor checks if it's not supported by the current compiler,
but I believe icpc defines some pre-processor flags the same as GCC for
compatibility.

SQLite does check for SQLITE_DISABLE_INTRINSIC to disable calling these,
regardless of compiler-defined pre-processor directives. I'd suggest adding
-DSQLITE_DISABLE_INTRINSIC to the build (I believe in src/Makevars
<https://github.com/rstats-db/RSQLite/blob/master/src/Makevars>).


... and the solution to the specific RSQLite issue is a  minor change to
the source code (see https://github.com/rstats-db/RSQLite/pull/224/files).


Below, instead, you can find the compilation log of sqlite with ICC under
Archlinux

==> Making package: sqlite 3.18.0-1 (sab  8 lug 2017, 16.52.07, CEST)
==> Checking runtime dependencies...
==> Checking buildtime dependencies...
==> Retrieving sources...
 -> Found sqlite-src-3180000.zip
 -> Found sqlite-doc-3180000.zip
 -> Found license.txt
==> Validating source files with sha1sums...
   sqlite-src-3180000.zip ... Passed
   sqlite-doc-3180000.zip ... Passed
   license.txt ... Passed
==> Extracting sources...
 -> Extracting sqlite-src-3180000.zip with bsdtar
 -> Extracting sqlite-doc-3180000.zip with bsdtar
==> Starting prepare()...
==> Removing existing $pkgdir/ directory...
==> Starting build()...
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking for gcc... icc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether icc accepts -g... yes
checking for icc option to accept ISO C89... none needed
checking for a sed that does not truncate output... /usr/bin/sed
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for fgrep... /usr/bin/grep -F
checking for ld used by icc... /usr/lib/hardening-wrapper/bin/ld
checking if the linker (/usr/lib/hardening-wrapper/bin/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 1572864
checking whether the shell understands some XSI constructs... yes
checking whether the shell understands "+="... yes
checking for /usr/lib/hardening-wrapper/bin/ld option to reload object
files... -r
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for ar... ar
checking for strip... strip
checking for ranlib... ranlib
checking command to parse /usr/bin/nm -B output from icc object... ok
checking how to run the C preprocessor... icc -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for dlfcn.h... yes
checking for objdir... .libs
checking if icc supports -fno-rtti -fno-exceptions... yes
checking for icc option to produce PIC... -fPIC -DPIC
checking if icc PIC flag -fPIC -DPIC works... yes
checking if icc static flag -static works... yes
checking if icc supports -c -o file.o... yes
checking if icc supports -c -o file.o... (cached) yes
checking whether the icc linker (/usr/lib/hardening-wrapper/bin/ld -m
elf_x86_64) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... no
checking for a BSD-compatible install... /usr/bin/install -c
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... no
checking for int8_t... yes
checking for int16_t... yes
checking for int32_t... yes
checking for int64_t... yes
checking for intptr_t... yes
checking for uint8_t... yes
checking for uint16_t... yes
checking for uint32_t... yes
checking for uint64_t... yes
checking for uintptr_t... yes
checking for sys/types.h... (cached) yes
checking for stdlib.h... (cached) yes
checking for stdint.h... (cached) yes
checking for inttypes.h... (cached) yes
checking malloc.h usability... yes
checking malloc.h presence... yes
checking for malloc.h... yes
checking for fdatasync... yes
checking for gmtime_r... yes
checking for isnan... yes
checking for localtime_r... yes
checking for localtime_s... no
checking for malloc_usable_size... yes
checking for strchrnul... yes
checking for usleep... yes
checking for utime... yes
checking for pread... ^[[Byes
checking for pread64... yes
checking for pwrite... yes
checking for pwrite64... yes
checking for tclsh8.6... tclsh8.6
configure: Version set to 3.18
configure: Release set to 3.18.0
configure: Version number set to 3018000
checking whether to support threadsafe operation... yes
checking for library containing pthread_create... -lpthread
checking for library containing pthread_mutexattr_init... none required
checking whether to support shared library linked as release mode or not...
no
checking whether to use an in-ram database for temporary tables... no
checking if executables have the .exe suffix... unknown
checking for Tcl configuration... found /usr/lib/tclConfig.sh
checking for existence of /usr/lib/tclConfig.sh... loading
checking for library containing readline... no
checking for library containing tgetent... no
checking for readline in -lreadline... no
checking readline.h usability... no
checking readline.h presence... no
checking for readline.h... no
checking for /usr/include/readline.h... no
checking for /usr/include/readline/readline.h... yes
checking for library containing fdatasync... none required
checking for library containing dlopen... none required
checking whether to support MEMSYS5... no
checking whether to support MEMSYS3... no
checking for library containing log... none required
checking for library containing log... (cached) none required
configure: creating ./config.status
config.status: creating Makefile
config.status: creating sqlite3.pc
config.status: creating config.h
config.status: config.h is unchanged
config.status: executing libtool commands
./libtool --mode=link icc -DSQLITE_SECURE_DELETE
-DSQLITE_MAX_VARIABLE_NUMBER=250000  -fast -pipe -gcc -DSQLITE_OS_UNIX=1
-I. -I/var/tmp/makepkg/sqlite/src/sqlite-src-3180000/src
-I/var/tmp/makepkg/sqlite/src/sqlite-src-3180000/ext/rtree
-I/var/tmp/makepkg/sqlite/src/sqlite-src-3180000/ext/icu
-I/var/tmp/makepkg/sqlite/src/sqlite-src-3180000/ext/fts3 -I/va
r/tmp/makepkg/sqlite/src/sqlite-src-3180000/ext/async
-I/var/tmp/makepkg/sqlite/src/sqlite-src-3180000/ext/session
-D_HAVE_SQLITE_CONFIG_H -DBUILD_sqlite -DNDEBUG -I/usr/include -
DSQLITE_THREADSAFE=1 -DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_FTS4
-DSQLITE_ENABLE_FTS5 -DSQLITE_ENABLE_JSON1 -DSQLITE_ENABLE_RTREE
-DSQLITE_ENABLE_COLUMN_METADATA=1 -DSQLITE_ENABLE_U
NLOCK_NOTIFY -DSQLITE_ENABLE_DBSTAT_VTAB=1 -DSQLITE_ENABLE_FTS3_TOKENIZER=1
  -Wl,-O1,--sort-common,--as-needed,-z,relro,--hash-style=gnu
 -DHAVE_READLINE=1 -I/usr/include/readlin
e -DHAVE_EDITLINE=0 -DSQLITE_ENABLE_JSON1 -DSQLITE_ENABLE_FTS4
-DSQLITE_ENABLE_EXPLAIN_COMMENTS -DSQLITE_ENABLE_UNKNOWN_SQL_FUNCTION -o
sqlite3 \
       /var/tmp/makepkg/sqlite/src/sqlite-src-3180000/src/shell.c sqlite3.c
\
        -lpthread   -rpath "/usr/lib"
libtool: link: icc -DSQLITE_SECURE_DELETE -DSQLITE_MAX_VARIABLE_NUMBER=250000
-fast -pipe -gcc -DSQLITE_OS_UNIX=1 -I.
-I/var/tmp/makepkg/sqlite/src/sqlite-src-3180000/src
-I/var/t
mp/makepkg/sqlite/src/sqlite-src-3180000/ext/rtree
-I/var/tmp/makepkg/sqlite/src/sqlite-src-3180000/ext/icu
-I/var/tmp/makepkg/sqlite/src/sqlite-src-3180000/ext/fts3 -I/var/tmp/ma
kepkg/sqlite/src/sqlite-src-3180000/ext/async
-I/var/tmp/makepkg/sqlite/src/sqlite-src-3180000/ext/session
-D_HAVE_SQLITE_CONFIG_H -DBUILD_sqlite -DNDEBUG -I/usr/include -DSQLITE_
THREADSAFE=1 -DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_FTS4 -DSQLITE_ENABLE_FTS5
-DSQLITE_ENABLE_JSON1 -DSQLITE_ENABLE_RTREE -DSQLITE_ENABLE_COLUMN_METADATA=1
-DSQLITE_ENABLE_UNLOCK_NO
TIFY -DSQLITE_ENABLE_DBSTAT_VTAB=1 -DSQLITE_ENABLE_FTS3_TOKENIZER=1 -Wl,-O1
-Wl,--sort-common -Wl,--as-needed -Wl,-z -Wl,relro -Wl,--hash-style=gnu
-DHAVE_READLINE=1 -I/usr/includ
e/readline -DHAVE_EDITLINE=0 -DSQLITE_ENABLE_JSON1 -DSQLITE_ENABLE_FTS4
-DSQLITE_ENABLE_EXPLAIN_COMMENTS -DSQLITE_ENABLE_UNKNOWN_SQL_FUNCTION -o
sqlite3 /var/tmp/makepkg/sqlite/sr
c/sqlite-src-3180000/src/shell.c sqlite3.c  -lpthread -Wl,-rpath
-Wl,/usr/lib
ipo: warning #11021: unresolved write_history
       Referenced in /tmp/ipo_iccqblmcn.o
ipo: warning #11021: unresolved stifle_history
       Referenced in /tmp/ipo_iccqblmcn.o
ipo: warning #11021: unresolved read_history
       Referenced in /tmp/ipo_iccqblmcn.o
ipo: warning #11021: unresolved add_history
       Referenced in /tmp/ipo_iccqblmcn.o
ipo: warning #11021: unresolved readline
       Referenced in /tmp/ipo_iccqblmcn.o
/tmp/ipo_iccqblmcn.o: In function `main':
ipo_out.c:(.text+0x19af): warning: Using 'getpwuid' in statically linked
applications requires at runtime the shared libraries from the glibc
version used for linking
/tmp/ipo_iccqblmcn.o: In function `unixDlOpen':
ipo_out.c:(.text+0x13e5b9): warning: Using 'dlopen' in statically linked
applications requires at runtime the shared libraries from the glibc
version used for linking
/tmp/ipo_iccqblmcn.o: In function `main':
ipo_out.c:(.text+0x1220): undefined reference to `read_history'
ipo_out.c:(.text+0x1242): undefined reference to `stifle_history'
ipo_out.c:(.text+0x124a): undefined reference to `write_history'
/tmp/ipo_iccqblmcn.o: In function `process_input':
ipo_out.c:(.text+0x4b840): undefined reference to `readline'
ipo_out.c:(.text+0x4b85a): undefined reference to `add_history'
/tmp/ipo_iccqblmcn.o: In function `sqlite3VdbeExec':
ipo_out.c:(.text+0x7dd71): undefined reference to `__builtin_add_overflow'
ipo_out.c:(.text+0x8135a): undefined reference to `__builtin_mul_overflow'
ipo_out.c:(.text+0x813d3): undefined reference to `__builtin_sub_overflow'
ipo_out.c:(.text+0x83e24): undefined reference to `__builtin_add_overflow'
/tmp/ipo_iccqblmcn.o: In function `sumStep':
ipo_out.c:(.text+0x12632d): undefined reference to `__builtin_add_overflow'
make: *** [Makefile:596: sqlite3] Error 1
==> ERROR: A failure occurred in build().
   Aborting...


Thank you very much for your attention.


R
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: cannot compile sqlite3 with intel compiler (ICC) - log and hints for a possible solution

Richard Hipp-3
On 7/8/17, Roberto C. <[hidden email]> wrote:
>
> The undefined symbol, __builtin_mul_overflow, is a GCC primitive. SQLite
> has the option to call it directly. It's supposed to disable itself through
> a some pre-processor checks if it's not supported by the current compiler,
> but I believe icpc defines some pre-processor flags the same as GCC for
> compatibility.
>

Please send me a complete listing of the icpc pre-defined symbols.
For GCC and clang, you obtain such a listing using a command like
this:

     gcc -dM -E -x c /dev/null

Maybe icpc works similarly.  You can send me the listing directly to
the email below, or post it on the mailing list - whichever is easiest
for you.

--
D. Richard Hipp
[hidden email]
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: cannot compile sqlite3 with intel compiler (ICC) - log and hints for a possible solution

Richard Hipp-3
In reply to this post by Roberto C.
On 7/8/17, Roberto C. <[hidden email]> wrote:
>
> while trying to install and sqlite package for R statistical language
> (Rsqlite)  I came across I bug: i could not compile the package with the
> intel compiler ICC / ICPC. You can find the whole discussion (and the
> solution) here - https://github.com/rstats-db/RSQLite/issues/223

(1) If I understand correctly, icpc is the C++ compiler and icc is the
C compiler. SQLite is written in C, not C++, so you should always use
icc not icpc for compiling SQLite.

(2) I downloaded and installed the 30-day free trial of ICC just now.
The trunk version of SQLite compiles and runs fine. "icc -v" reports
"icc version 17.0.4". Maybe this issue only applies to older versions
of icc?

(3) I compiled SQLite on each of gcc-5.4, gcc-7.1, clang-3.5, and
icc-17.0 and compared both the size of the resulting binary and the
performance. icc gave the largest binary and the slowest performance.
Here are the actual results:

gcc-5.4: 491585 bytes, 1,124 million CPU cycles.
gcc-7.1: 487582 bytes, 1,121 million CPU cycles.
clang-3.5: 569570 bytes, 1,170 million CPU cycles
icc-17.0: 536596 bytes, 1,274 million CPU cycles

As you can see, the gcc-compiled binary of SQLite is 9% smaller and
12% faster than the icc-compiled binary. So, maybe the solution is to
just not use icc?


--
D. Richard Hipp
[hidden email]
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: cannot compile sqlite3 with intel compiler (ICC) - log and hints for a possible solution

Bob Friesenhahn
On Sat, 8 Jul 2017, Richard Hipp wrote:

>
> (3) I compiled SQLite on each of gcc-5.4, gcc-7.1, clang-3.5, and
> icc-17.0 and compared both the size of the resulting binary and the
> performance. icc gave the largest binary and the slowest performance.
> Here are the actual results:
>
> gcc-5.4: 491585 bytes, 1,124 million CPU cycles.
> gcc-7.1: 487582 bytes, 1,121 million CPU cycles.
> clang-3.5: 569570 bytes, 1,170 million CPU cycles
> icc-17.0: 536596 bytes, 1,274 million CPU cycles
>
> As you can see, the gcc-compiled binary of SQLite is 9% smaller and
> 12% faster than the icc-compiled binary. So, maybe the solution is to
> just not use icc?

Sqlite3 has been cycle-optimized for GCC.  You now have 29-days to
also cycle-optimize it for ICC.

Bob
--
Bob Friesenhahn
[hidden email], http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: cannot compile sqlite3 with intel compiler (ICC) - log and hints for a possible solution

Jens Alfke-2

> On Jul 13, 2017, at 7:49 AM, Richard Hipp <[hidden email]> wrote:
>
> I've spent some time investigating this, and I think the answer is
> that GCC simply generates better code.

Have you tested Clang? These days it may generate even better code, especially with -Ofast and the link-time optimizer. (SQLite itself won’t benefit from LTO because it’s already agglomerated before being compiled, but LTO allows SQLite code to be inlined into calling functions in the client program.)

—Jens
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: cannot compile sqlite3 with intel compiler (ICC) - log and hints for a possible solution

Jens Alfke-2
In reply to this post by Roberto C.

> On Jul 13, 2017, at 3:41 PM, Keith Medcalf <[hidden email]> wrote:
>
> CLang must be just catching up.  
>
> That is better though than Intel, Microsoft, et al, who make really shoddy compilers (buggy and suitable for amateur use only) let alone have any actual "advanced" features.

Why the chip on your shoulder? I don’t recall asking for a history lesson, let alone a snarky one. I simply asked how SQLite’s code built with Clang compared with GCC.

—Jens
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Loading...