VFS

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

VFS

Robert Hairgrove
I'm trying to understand how the VFS implementation works. If I fetch
the default VFS with sqlite3_vfs_find(NULL), these members:

  xDlOpen
  xDlError
  xDlSym
  xDlClose

all have non-NULL values. However, if I open a database and inspect the
VFS contained in the sqlite3*, these four members are all NULL. Which
function resets them, and why? I have no extensions loaded, but
SQLITE_OMIT_EXTENSIONS is also not defined.

Thanks.

_______________________________________________
sqlite-users mailing list
[hidden email]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: VFS

Roger Binns
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 02/22/2011 05:29 AM, Robert Hairgrove wrote:
> I'm trying to understand how the VFS implementation works.

What you think you are seeing is not happening.  The documentation is correct.

> However, if I open a database and inspect the VFS contained in the sqlite3*,

That should just point to the VFS used.  There is no copying or modification.

Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk1j6FkACgkQmOOfHg372QTR8gCgsX+dYtQFkeYRZDskwKoQRXdc
eGUAoJqhFJYdkOp40Usuujws3KSFMT2m
=kYzg
-----END PGP SIGNATURE-----
_______________________________________________
sqlite-users mailing list
[hidden email]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: VFS

Robert Hairgrove
On Tue, 2011-02-22 at 08:46 -0800, Roger Binns wrote:
> On 02/22/2011 05:29 AM, Robert Hairgrove wrote:
> > I'm trying to understand how the VFS implementation works.
>
> What you think you are seeing is not happening.  The documentation is correct.

OK ... but what about that which the GDB debugger is seeing?

> > However, if I open a database and inspect the VFS contained in the sqlite3*,
>
> That should just point to the VFS used.  There is no copying or modification.

Or so says the documentation...

Maybe some more details will help here:

The OS is Linux Ubuntu 10.04 LTS, AKA "Lucid Lynx" running on a Dell
Inspiron 1420N notebook (and Dell supplies the drivers etc. for this
distro of Linux). GCC version is 4.4.3.

SQLite version is 3.6.19 (I know it's not the latest and greatest, but
this is what is supplied together with the Qt sources... these are
version 4.7.1 -- which is pretty much up-to-date, but not cutting edge).

I compiled the Qt libraries from source myself and built SQLite
statically into Qt, as opposed to a plug-in.  The SQLite sources are all
in the amalgamation file "sqlite.c".

I created a little console project in QtCreator to test the SQLite VFS
functionality. I had to add "sqlite3.c" to the project because I was
getting linker errors when trying to call some of the SQLite API
functions, such as "sqlite3_vfs_find" ... (hmmm...seems that these
symbols aren't exported from the Qt shared libraries).

Obviously, SOMETHING is resetting these pointers. Here is a snippet from
the test code I wrote. It should compile and run OK if you have a newer
version of Qt installed on your system. Here is my Qt .pro file:

//==== my Qt project file: =======================================
QT += core sql
INCLUDEPATH += \
  {wherever the Qt sources are...}/src/3rdparty/sqlite
SOURCES += main.cpp \
  {wherever the Qt sources are...}/src/3rdparty/sqlite/sqlite3.c
HEADERS += \
  {wherever the Qt sources are...}/src/3rdparty/sqlite/sqlite3.h
//==== end of Qt project file =======================================

And here is the source code of my test app in "main.cpp":

//==== BEGIN CODE: ============================================
#include <QtCore>
#include <QtSql>
#include <iostream>
#include <sqlite3.h>

void showVFS(const sqlite3_vfs * const &);

int main(int argc, char *argv[])
{
    QSqlDatabase db = QSqlDatabase::addDatabase("QSQLITE","default_db");
    sqlite3 * pSqliteDb = 0;
    //----------------------------------------------------------------
    // Using an on-disk database seems to give the exact same results:
    // db.setDatabaseName("/home/bob/code/SQLite_Test_VFS/test.db");
    //----------------------------------------------------------------
    db.setDatabaseName(":memory:");
    if (db.open()) {
        //-------------------------------------------------------
        // Some of the following lines of code are copied and
        // pasted from the Qt docs:
        //-------------------------------------------------------
        QVariant v = db.driver()->handle();
        if (v.isValid() && qstrcmp(v.typeName(), "sqlite3*")==0) {
            // v.data() returns a pointer to the handle...
            //????????????????????????????????????????????????????
            // Actually, it seems to return either int
            // or void* ... (see below):
            //????????????????????????????????????????????????????
            sqlite3 *pSqliteDb = *static_cast<sqlite3 **>(v.data());
            if (pSqliteDb) {
                //????????????????????????????????????????????????????
                // Interesting that using static_cast<> does not work below;
                // probably because QSqlDriver::handle() returns void* (???)
                // although the Qt docs AND the sources say differently:
                //????????????????????????????????????????????????????
                sqlite3_vfs * pvfs = *reinterpret_cast<sqlite3_vfs **>(pSqliteDb);
                //----------------------------------------------------------------
                // The line below gives a VFS with non-NULL values
                // for the struct members under discussion:
                //----------------------------------------------------------------
                // sqlite3_vfs * pvfs = sqlite3_vfs_find(0);
                //----------------------------------------------------------------
                if (pvfs) {
                    showVFS(pvfs);
                }
            } else {
                std::cout << "No VFS handle!" << std::endl;
            }
        }
    }
    return 0;
}
////////////////////////////////////////////////////////
void showVFS(const sqlite3_vfs * const &pVfs)
{
    if (pVfs->zName) {
        std::cout << "=====================================================\n";
        std::cout << "Name of VFS:\t" << pVfs->zName << std::endl;
#ifdef SQLITE_OMIT_LOAD_EXTENSION
        std::cout << "SQLITE_OMIT_LOAD_EXTENSION = true" << std::endl;
#endif
        std::cout << "=====================================================\n";
        std::cout << "iVersion        = " << pVfs->iVersion   << std::endl;            /* Structure version number */
        std::cout << "szOsFile        = " << pVfs->szOsFile   << std::endl;            /* Size of subclassed sqlite3_file */
        std::cout << "mxPathname      = " << pVfs->mxPathname << std::endl;          /* Maximum file pathname length */
        std::cout << "pNext           = " << pVfs->pNext      << std::endl;       /* Next registered VFS */
        std::cout << "zName           = " << pVfs->zName      << std::endl;       /* Name of this virtual file system */
        std::cout << "pAppData        = " << pVfs->pAppData   << std::endl;          /* Pointer to application-specific data */

        std::cout << "xOpen           = " << (void*)pVfs->xOpen         << std::endl;
        std::cout << "xDelete         = " << (void*)pVfs->xDelete       << std::endl;
        std::cout << "xAccess         = " << (void*)pVfs->xAccess       << std::endl;
        std::cout << "xFullPathname   = " << (void*)pVfs->xFullPathname << std::endl;
        /////////////////////////////////////////////////////////////////////////////
        // Up to here, nothing is changed in the VFS whether I use sqlite3_vfs_find()
        // or get it from the sqlite3* database pointer. But the following four
        // members are always NULL if I use the VFS in the database which is open!...
        //---------------------------------------------------------------------------
        std::cout << "xDlOpen         = " << (void*)(pVfs->xDlOpen)     << std::endl;
        std::cout << "xDlError        = " << (void*)(pVfs->xDlError)    << std::endl;
        std::cout << "xDlSym          = " << (void*)(pVfs->xDlSym)      << std::endl;
        std::cout << "xDlClose        = " << (void*)(pVfs->xDlClose)    << std::endl;
        /////////////////////////////////////////////////////////////////////////////
        std::cout << "xRandomness     = " << (void*)pVfs->xRandomness   << std::endl;
        std::cout << "xSleep          = " << (void*)pVfs->xSleep        << std::endl;
        std::cout << "xCurrentTime    = " << (void*)pVfs->xCurrentTime  << std::endl;
        std::cout << "xGetLastError   = " << (void*)pVfs->xGetLastError << std::endl;
    }
}
//==== END CODE ============================================


_______________________________________________
sqlite-users mailing list
[hidden email]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: VFS

Roger Binns
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 02/22/2011 02:39 PM, Robert Hairgrove wrote:
> On Tue, 2011-02-22 at 08:46 -0800, Roger Binns wrote:
>> What you think you are seeing is not happening.  The documentation is correct.
>
> OK ... but what about that which the GDB debugger is seeing?

There are two possibilities:

1 - The SQLite code deliberately (against documented behaviour) goes around
copying VFS structures, and sneaks in behind the scenes to zero out some
members all the while craftily hiding how this is done so that searching the
source code won't find it.  It also hides the previous values so that
extension loading still works.

2 - When layers of third party code are compiled, linked, some static, some
dynamic, and combined in a process, you and/or gdb get confused.

For Linux the only way the default VFS ends up with zero for the dl
functions is if it is compiled with SQLITE_OMIT_LOAD_EXTENSION (which
incidentally has often been the default for many maintainers).

Other things I have seen are multiple copies of SQLite getting loaded into
the same process.  This is especially an issue on Mac where CoreData picks
up the system copy.  The copies all coexist but will confuse you.

I have been caught out by shared library unloading and then reloading.
Because Linux uses ASLR the reload happens at a different address but any
registrations made during the first load point to the wrong addresses.

Roger
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk1khwwACgkQmOOfHg372QRvkACgvIRRBiQV3k9suHC38EhYEKQ8
UmsAnRmXUyam/B2FfiTiS3/cEGyLv/rH
=1gl3
-----END PGP SIGNATURE-----
_______________________________________________
sqlite-users mailing list
[hidden email]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: VFS

Robert Hairgrove
It appears that Qt (or more accurately, WebKit) had defined
SQLITE_OMIT_LOAD_EXTENSION and a couple of other symbols at compile
time, and therefore when Qt opens a database, it uses a slightly
different VFS than the default VFS contained in sqlite3.c (i.e., no xDl*
members are set).

I believe now that I should recompile Qt and make SQLite a plug-in ...
or else build it separately and let Qt use the system's SQLite instead
of their version.

_______________________________________________
sqlite-users mailing list
[hidden email]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users