Database identified as zip

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Database identified as zip

Dingyuan Wang
Dear list,

When my database opened with sqlite3 command line program, it shows this:

SQLite version 3.26.0 2018-10-29 21:01:28
Enter ".help" for usage hints.
sqlite> .schema
CREATE VIRTUAL TABLE zip USING zipfile('funddata.db')
/* zip(name,mode,mtime,sz,rawdata,data,method) */;
sqlite> select * from zip;
Error: failed to read CDS at offset 119844

When I use python to open this database, it reads correctly.

There are a lot of zip files stored in this "datasave" table, where v is
the zip file.

CREATE TABLE datasave (k TEXT PRIMARY KEY, ver INTEGER, v BLOB)

I look at the database file, there is a half zip file in the end. So
maybe the PK header confused the SQLite command line.

Is this considered a bug?
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Database identified as zip

Richard Hipp-3
On 10/30/18, Dingyuan Wang <[hidden email]> wrote:

> I look at the database file, there is a half zip file in the end. So
> maybe the PK header confused the SQLite command line.

That is the likely explanation.

The code that deduces the database type (whether it is an ordinary
SQLite database file, or a ZIP archive, or something else) is seen at
(https://www.sqlite.org/src/artifact/ac4a731dac549746?ln=3666-3694).
Line 3687 is the one that is causing problems for you, I am guessing.

>
> Is this considered a bug?

I suppose it is.  I'm still trying to figure out exactly how to fix
it, though.  Perhaps we should only look for the EOCD record at the
end of the file if the file does not begin with the proper SQLite
database prefix.
--
D. Richard Hipp
[hidden email]
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Database identified as zip

Stephen Chrzanowski
First off, I didn't know that SQLite could read from zip files directly.  I
don't like it, but, I'm sure there's a use case somewhere.

Second, shouldn't SQLite only consider reading a file if the file size is
zero or if the appropriate SQLite header is found, and completely ignore
the file extension?  If I create a normal SQLite file, then rename the file
to a .zip, the content isn't a ZIP, so things would break, I would think.


On Tue, Oct 30, 2018 at 9:49 AM Richard Hipp <[hidden email]> wrote:

> On 10/30/18, Dingyuan Wang <[hidden email]> wrote:
>
> > I look at the database file, there is a half zip file in the end. So
> > maybe the PK header confused the SQLite command line.
>
> That is the likely explanation.
>
> The code that deduces the database type (whether it is an ordinary
> SQLite database file, or a ZIP archive, or something else) is seen at
> (https://www.sqlite.org/src/artifact/ac4a731dac549746?ln=3666-3694).
> Line 3687 is the one that is causing problems for you, I am guessing.
>
> >
> > Is this considered a bug?
>
> I suppose it is.  I'm still trying to figure out exactly how to fix
> it, though.  Perhaps we should only look for the EOCD record at the
> end of the file if the file does not begin with the proper SQLite
> database prefix.
> --
> D. Richard Hipp
> [hidden email]
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Database identified as zip

Richard Hipp-3
In reply to this post by Dingyuan Wang
On 10/30/18, Dingyuan Wang <[hidden email]> wrote:
>
> Is this considered a bug?

Should be fixed on trunk, now.
--
D. Richard Hipp
[hidden email]
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Database identified as zip

Richard Hipp-3
In reply to this post by Stephen Chrzanowski
On 10/30/18, Stephen Chrzanowski <[hidden email]> wrote:
>
> Second, shouldn't SQLite only consider reading a file if the file size is
> zero or if the appropriate SQLite header is found, and completely ignore
> the file extension?

The core SQLite library understands SQLite database files only.
Reading and writing of ZIP archives and other formats is handled by
extensions.  Those extensions are built into the CLI.  The following
discussion applies to the CLI, not to the core SQLite library.

The file extension is ignored, except when creating a new file, in
which case the presence of the ".zip" extension will force it to be a
ZIP archive if no other mode is specified.

When opening an existing database file, the followings rules apply in
the order shown:

(1) Honor command-line switches such as --zip or --append.

(2) If the file begins with the SQLite header, then treat it as an
ordinary SQLite database file.

(3) If the file ends with the ZIP EOCD record, treat it as a ZIP archive.

(4) If the file ends with the AppendVFS tail record, treat it as an
AppendVFS file.

(5) Otherwise treat the file as an ordinary SQLite database.

The bug was that step (2) was omitted.  That bug is fixed with
check-in https://www.sqlite.org/src/info/7989bbda70a24611

--
D. Richard Hipp
[hidden email]
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Database identified as zip

Stephen Chrzanowski
That said, I still didn't know the shell/CLI could do that.  But 14.5 and
14.6 are pretty far down https://sqlite.org/cli.html so, I probably gave up
after 6.1 or 6.2.... {smirk}

Good nugget to keep rattling in my head.

On Tue, Oct 30, 2018 at 10:44 AM Richard Hipp <[hidden email]> wrote:

>
> The core SQLite library understands SQLite database files only.
> Reading and writing of ZIP archives and other formats is handled by
> extensions.  Those extensions are built into the CLI.  The following
> discussion applies to the CLI, not to the core SQLite library.
>
>
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Database identified as zip

Dingyuan Wang
In reply to this post by Richard Hipp-3
It opens correctly now.

2018-10-30 22:37, Richard Hipp:
> On 10/30/18, Dingyuan Wang <[hidden email]> wrote:
>>
>> Is this considered a bug?
>
> Should be fixed on trunk, now.
>
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users