What if OpenDocument were built atop SQLite?

classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|

What if OpenDocument were built atop SQLite?

Richard Hipp-3
See the essay at:

   http://www.sqlite.org/affcase1.html

Comments, criticism, and feedback are welcomed.
--
D. Richard Hipp
[hidden email]
_______________________________________________
sqlite-users mailing list
[hidden email]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: What if OpenDocument were built atop SQLite?

John McKown
On Mon, Sep 8, 2014 at 7:35 AM, Richard Hipp <[hidden email]> wrote:
> See the essay at:
>
>    http://www.sqlite.org/affcase1.html
>
> Comments, criticism, and feedback are welcomed.
> --
> D. Richard Hipp
> [hidden email]

A very good presentation of an good idea. Although I am not a
developer or historian of ODP, it is basically an offshoot of
OpenOffice. Which may explain why the output is in zip format.
Open/Libre Office is Java based. And Java has always had zip file
processing integrated into its standard library. I guess that the
developers wanted to just use what was universally available, and not
depend on other "3rd party" libraries, such as SQLite.

The above does not mean that I think it should not be considered, but
it does mean that, to an extent, Open/Libre Office would be bound to
SQLite as well as Java. Which the developers may not want to do. I do
hope that they would consider it. Of course, this would either mean
"dual path" in the code for zip ODP files vs. SQLite ODP files or a
complete break with compatibility with the old file structure. Which
may be "too Microsoft-ish" for them.


--
There is nothing more pleasant than traveling and meeting new people!
Genghis Khan

Maranatha! <><
John McKown
_______________________________________________
sqlite-users mailing list
[hidden email]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: What if OpenDocument were built atop SQLite?

Teg-3
In reply to this post by Richard Hipp-3
Hello Richard,

Monday, September 8, 2014, 8:35:30 AM, you wrote:

RH> See the essay at:

RH>    http://www.sqlite.org/affcase1.html

RH> Comments, criticism, and feedback are welcomed.

I  use  Sqlite  as  a  container  for  images because it's superior to
CBR/CBZ  files (which are rar and zip files).  I think this is a great
idea.


>>Finally, let us reiterate that this essay is a thought experiment. The
>>OpenDocument format is well-established and already well-designed.
>>Nobody really believes that OpenDocument should be changed to use
>>SQLite as its container instead of ZIP. Nor is this article a
>>criticism of OpenDocument for not choosing SQLite as its container
>>since OpenDocument predates SQLite. Rather, the point of this article
>>is to use OpenDocument as a concrete example of how SQLite can be used
>>to build better application file formats for future projects.

It  might  be too late in the game but, I would have preferred this to
zips for sure.


Teg mailto:[hidden email]

_______________________________________________
sqlite-users mailing list
[hidden email]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: What if OpenDocument were built atop SQLite?

Simon Slavin-3
In reply to this post by Richard Hipp-3

On 8 Sep 2014, at 1:35pm, Richard Hipp <[hidden email]> wrote:

> See the essay at:
>
>   http://www.sqlite.org/affcase1.html

You would need to provide some short and simple 'viewer' code which can be used to extract a 'Quick View' or 'Thumbnail' file from the database, so the GUI Shell can show it to the user when they're browsing files.  Generally speaking you never modify just a single file in such a document because you need to modify the thumbnail file when any change is made to the content.

Given that SQLite code is public anyway, and this would need no writing and no locking, it wouldn't be hard, but it would need to be done.

Also I doubt your compression figures are accurate for big files.  A typical .docx document on my computer is 32K long but expands to 87K unzipped plus 576K for the 'Quick View' file.  But I don't have to time to look into this in detail.

Simon.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: What if OpenDocument were built atop SQLite?

Roger Binns
In reply to this post by Richard Hipp-3
On 08/09/14 05:35, Richard Hipp wrote:
> See the essay at:
>
>    http://www.sqlite.org/affcase1.html
>
> Comments, criticism, and feedback are welcomed.

BTW historically Microsoft used a "file system" for Office files before the
XML stuff (ie even in the first versions from over 20 years ago).  Back then
the file system was a variant of FAT.  The underlying format of the files
was also optimised for modifications, recording things as a series of chunks
rather than as contiguous content.  The reason I mention all this is because
the obvious solution for an XML world is to do the ZIP file approach
allowing much compatibility for existing code and formats, not because it is
the best solution out there.

"Incremental update is hard"

That isn't strictly true.  ZIP files (unlike the majority of formats) store
the central directory of content at the end of the file.  You can update a
zip file by appending the updated content, and then appending the new
central directory but with the file name pointing to the updated content not
the older version.  That does then require some form of garbage collection,
but space is way larger these days.  It also seems possible to leave space
after a file before the next one as room to grow thereby not needing a full
rewrite on small changes.

As to SQLite, I think the biggest issue is dealing with corrupted content.
A zip file has a file as a unit of content with each one separately
compressed and checksummed.  If you corrupted a random byte of a zip file
you could easily determine that it has happened and isolate which file is
affected.  You may even be able to recover from it (eg it was a small picture).

SQLite has no way of finding corrupted content nor isolating it.  The
integrity check pragma might find something if the corruption happened
somewhere in metadata, but other than that you are unlikely to discover it.
 Previous riding of that high horse (rejected):

  http://www.sqlite.org/src/tktview?name=72b01a982a

Zip also has encryption as standard which SQLite doesn't.

Overall I'd suggest keeping most of the content but changing the focus to
"Best practises when using SQLite as an application file format".  ie if
someone is sold on SQLite as the format then what should they do.  As an
example I have found that versioning and undo/redo are the most important
things to get right up front.  Showing the triggers to use for undo/redo
would be helpful.  Suggesting a thumbnail entry is good too, etc.

Roger
_______________________________________________
sqlite-users mailing list
[hidden email]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: What if OpenDocument were built atop SQLite?

Nico Williams
In reply to this post by Richard Hipp-3
And now that SQLite3 has recursive queries, building a mapping of
XML->SQL is relatively easy, which might make it even easier to switch
to SQLite3.

(Speaking of which, a XPath to SQL compiler would be really nice.  I
haven't sat down to think about whether that'd be feasible, but my
impression is that it should be.  An XSLT interpreter that used SQL
should be feasible as well.)

There's also your UnQL, no?
_______________________________________________
sqlite-users mailing list
[hidden email]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: What if OpenDocument were built atop SQLite?

Nico Williams
It'd also be good to have JSON and XML support, possibly as an
loadable extension.  That way XPath and similar expressions matching
document snippets in SQL string values could be used in SQL queries.

I've played with building a JSON extension for SQLite3 using jq's
excellent JSON C library.  The biggest "problem" with that work is
that the extension has to serialize values to JSON (and, of course,
parse) in many cases where it could be avoided with some help from
SQLite3.

It'd be *very* convenient if SQLite3 had a value type that corresponds
to loadable extensions' private types, and a protocol for releasing
and serializing values of such types.

(jq has an XPath-like language, but for JSON.)

Nico
--
_______________________________________________
sqlite-users mailing list
[hidden email]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: What if OpenDocument were built atop SQLite?

Simon Slavin-3

On 8 Sep 2014, at 11:45pm, Nico Williams <[hidden email]> wrote:

> It'd also be good to have JSON and XML support, possibly as an
> loadable extension.  That way XPath and similar expressions matching
> document snippets in SQL string values could be used in SQL queries.

Not so sure about that.  There are JSON and XML extensions available for pretty-much every language out there.  Do you feel they need to be tightly integrated with SQLite somehow ?

> It'd be *very* convenient if SQLite3 had a value type that corresponds
> to loadable extensions' private types, and a protocol for releasing
> and serializing values of such types.

Ah, but there you struck gold.  If this is to be adopted, SQLite needs a DataType of FILE. Complete with functions to do things like get a file's name, path, length, contents, and probably half a dozen other things that don't come to mind right now.  In fact this might be useful for SQLite even if it isn't going to be used as an OpenDoc file container.

Simon.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: What if OpenDocument were built atop SQLite?

Nico Williams
On Mon, Sep 8, 2014 at 6:16 PM, Simon Slavin <[hidden email]> wrote:
> On 8 Sep 2014, at 11:45pm, Nico Williams <[hidden email]> wrote:
>> It'd also be good to have JSON and XML support, possibly as an
>> loadable extension.  That way XPath and similar expressions matching
>> document snippets in SQL string values could be used in SQL queries.
>
> Not so sure about that.  There are JSON and XML extensions available for pretty-much every language out there.  Do you feel they need to be tightly integrated with SQLite somehow ?

No, I don't.  I rather dislike the way Postgres did the hstore and the
subsequent JSON support.

I should clarify: what I don't like is ad-hoc syntax.  I would like a
way to embed arbitrary external languages like jq or XPath in SQL
expressions.

>> It'd be *very* convenient if SQLite3 had a value type that corresponds
>> to loadable extensions' private types, and a protocol for releasing
>> and serializing values of such types.
>
> Ah, but there you struck gold.  If this is to be adopted, SQLite needs a DataType of FILE. Complete with functions to do things like get a file's name, path, length, contents, and probably half a dozen other things that don't come to mind right now.  In fact this might be useful for SQLite even if it isn't going to be used as an OpenDoc file container.

Yes.  Pretty please, with sugar on top?

Nico
--
_______________________________________________
sqlite-users mailing list
[hidden email]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: What if OpenDocument were built atop SQLite?

John McKown
On Mon, Sep 8, 2014 at 6:32 PM, Nico Williams <[hidden email]> wrote:

> On Mon, Sep 8, 2014 at 6:16 PM, Simon Slavin <[hidden email]> wrote:
>> On 8 Sep 2014, at 11:45pm, Nico Williams <[hidden email]> wrote:
>>> It'd also be good to have JSON and XML support, possibly as an
>>> loadable extension.  That way XPath and similar expressions matching
>>> document snippets in SQL string values could be used in SQL queries.
>>
>> Not so sure about that.  There are JSON and XML extensions available for pretty-much every language out there.  Do you feel they need to be tightly integrated with SQLite somehow ?
>
> No, I don't.  I rather dislike the way Postgres did the hstore and the
> subsequent JSON support.
>
> I should clarify: what I don't like is ad-hoc syntax.  I would like a
> way to embed arbitrary external languages like jq or XPath in SQL
> expressions.
>
>>> It'd be *very* convenient if SQLite3 had a value type that corresponds
>>> to loadable extensions' private types, and a protocol for releasing
>>> and serializing values of such types.
>>
>> Ah, but there you struck gold.  If this is to be adopted, SQLite needs a DataType of FILE. Complete with functions to do things like get a file's name, path, length, contents, and probably half a dozen other things that don't come to mind right now.  In fact this might be useful for SQLite even if it isn't going to be used as an OpenDoc file container.
>
> Yes.  Pretty please, with sugar on top?
>
> Nico
> --

Hum, why not a URI data type instead? ref:
http://en.wikipedia.org/wiki/URI_scheme or
http://www.ietf.org/rfc/rfc3986.txt .This is a "proper superset" of a
URL. And it could encompass many other "access methods". Some that are
implemented in browsers are things like http://some.webpage.html,
file:///path/to/local/file, mailto:[hidden email],
ftp://user:[hidden email]/subdir/file and lots of other
possibilities. The RFC does not attempt to detail all possible scheme
(the first thing in the name, before the first :). SQLite could
implement the fetching of contents by doing something like loading a
dynamic library at run-time which might look like: lib<scheme>.so or
.dll Where <scheme> is the URI scheme. E.g. libftp.so or libhttp.so .
If just used in a SELECT, SQLite would just return a string value
equal to the value such as you might see in a browser. And if the
contents were needed,  a function such as fetch() might be use. E.g.
SELECT FETCH(urifield) FROM table; would get the URI for each non-NULL
value of "urifield", inspect the first portion for the scheme, load
the proper library and then pass the URI as a string value to that
routine. Of course, this is getting fairly involved. And what is done
if such a library does not exist? Error out the entire SELECT? Return
a NULL? It gets very complicated. And I'm sure Dr. Hipp will see more
problems and implications that I could ever imagine.


--
There is nothing more pleasant than traveling and meeting new people!
Genghis Khan

Maranatha! <><
John McKown
_______________________________________________
sqlite-users mailing list
[hidden email]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: What if OpenDocument were built atop SQLite?

Nico Williams
On Mon, Sep 8, 2014 at 8:05 PM, John McKown
<[hidden email]> wrote:
> Hum, why not a URI data type instead? ref:

Because we're talking about a purely internal type, with internal
linkage.  Externally it would appear as TEXT or BLOB.

You'd use the TEXT type to store JSON, XML, ..., BLOB to store binary
encodings.  Internally you need to parse and serialize these things at
the boundaries.  If the boundaries are the existing ones then you end
up parsing and serializing more than if the boundaries were just the
file format.

What I've got in mind is an extension to the loadable extension
framework that allows each extension to add one (or more)
pseudo-type(s) that have: a parser, a serializer, a copy/incref, and a
release/decref operations, and probably nothing else.  Those
operations would be the internal linkage.

API-wise there would be new statement parameter binding and cursor
column accessors for using pseudo-types instead of TEXT/BLOB, but if
these aren't used you'd get TEXT/BLOB.

Language-wise there'd be not much special: virtual tables and
user-defined functions are enough, really.

If you ran a SELECT in the sqlite3 shell you'd get TEXT or BLOB
results, but if you ran it using the C API you could access the
internal extension type (e.g., the C 'jv' type provided by libjq).

Nico
--
_______________________________________________
sqlite-users mailing list
[hidden email]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: What if OpenDocument were built atop SQLite?

Keith Medcalf
In reply to this post by John McKown

Not really very difficult, and it does not require another type.  You just need a scalar function to process the URI passed to the function and return the result -- just like the readfile() (in the fileio.c extension, or included in the shell) function does for a "file-specifier" which can be opened by the platform open api.

Of course, the size of the module required to do this might be rather large.  Or it could be very simple.  Here is a simple example in python using apsw and fs (output truncated in various places ...) with no meaningful results (ie, null) if the URI doesn't parse:

--- apswopener.py ---
import apsw
from fs.opener import opener
import sys

def uricontents(URI):
    if not URI:
        return
    try:
        contents = opener.getcontents(URI)
    except:
        return None
    try:
        return unicode(contents)
    except:
        return bytearray(contents)

def defineuricontents(cn):
    cn.createscalarfunction('uricontents', uricontents, 1)

apsw.connection_hooks.append(defineuricontents)

db = None
if len(sys.argv) > 1:
    db = apsw.Connection(sys.argv[1])
apsw.Shell(db=db).cmdloop()
--- EOF ---

2014-09-08 19:59:08 [D:\Data\Apps\fs]
>apswopener.py
SQLite version 3.8.7 (APSW 3.8.5-r1)
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> select uricontents('d:\source\sqlite\x');
SQLITE_32BIT_ROWID
SQLITE_4_BYTE_ALIGNED_MALLOC
SQLITE_64BIT_STATS
SQLITE_ALLOW_COVERING_INDEX_SCAN
[... pasting truncated ...]

sqlite> select uricontents('http://mail.dessus.com/');

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html lang="en" dir="LTR">
<head>
        <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />

        <title> CommuniGate Pro dessus.com Entrance</title>
        <link rel="stylesheet" href="/SkinFiles/dessus.com//style.css" type="text/css" />
  <meta http-equiv="x-dns-prefetch-control" content="off" />
[... pasting truncated ...]

sqlite> select uricontents('http://www.sqlite.org');
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html><head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>SQLite Home Page</title>
<style type="text/css">
body {
    margin: auto;
    font-family: Verdana, sans-serif;
    padding: 8px 1%;
}

a { color: #044a64 }
a:visited { color: #734559 }
[... pasting truncated ...]

sqlite> select length(uricontents('http://www.sqlite.org/favicon.ico'));
318
sqlite> select uricontents('http://www.sqlite.org/favicon.ico');
<Binary data>
sqlite> select hex(uricontents('http://www.sqlite.org/favicon.ico'));
000001000100101010000100040028010000160000002800000010000000200000000100040000000000000000000000000000000000000000000000000000000000A8A8A800
98A47800705050000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001322222222222231322222022002
00232222220220202222222222022020000220002202202022022222220100020022222222222022222222222202202222222222222222202222200022200002000000200200
20020022220002002202002200002200220200220022220020020022300002200022002313222222222222310000000000000000000000000000000000000000000000000000
0000000000000000000000000000000000000000000000000000000000000000000000000000
sqlite> select typeof(uricontents('http://www.sqlite.org/favicon.ico'));
blob
sqlite> select typeof(uricontents('http://www.sqlite.org/'));
text

>-----Original Message-----
>From: [hidden email] [mailto:sqlite-users-
>[hidden email]] On Behalf Of John McKown
>Sent: Monday, 8 September, 2014 19:06
>To: General Discussion of SQLite Database
>Subject: Re: [sqlite] What if OpenDocument were built atop SQLite?
>
>On Mon, Sep 8, 2014 at 6:32 PM, Nico Williams <[hidden email]>
>wrote:
>> On Mon, Sep 8, 2014 at 6:16 PM, Simon Slavin <[hidden email]>
>wrote:
>>> On 8 Sep 2014, at 11:45pm, Nico Williams <[hidden email]>
>wrote:
>>>> It'd also be good to have JSON and XML support, possibly as an
>>>> loadable extension.  That way XPath and similar expressions matching
>>>> document snippets in SQL string values could be used in SQL queries.
>>>
>>> Not so sure about that.  There are JSON and XML extensions available
>for pretty-much every language out there.  Do you feel they need to be
>tightly integrated with SQLite somehow ?
>>
>> No, I don't.  I rather dislike the way Postgres did the hstore and the
>> subsequent JSON support.
>>
>> I should clarify: what I don't like is ad-hoc syntax.  I would like a
>> way to embed arbitrary external languages like jq or XPath in SQL
>> expressions.
>>
>>>> It'd be *very* convenient if SQLite3 had a value type that
>corresponds
>>>> to loadable extensions' private types, and a protocol for releasing
>>>> and serializing values of such types.
>>>
>>> Ah, but there you struck gold.  If this is to be adopted, SQLite needs
>a DataType of FILE. Complete with functions to do things like get a
>file's name, path, length, contents, and probably half a dozen other
>things that don't come to mind right now.  In fact this might be useful
>for SQLite even if it isn't going to be used as an OpenDoc file
>container.
>>
>> Yes.  Pretty please, with sugar on top?
>>
>> Nico
>> --
>
>Hum, why not a URI data type instead? ref:
>http://en.wikipedia.org/wiki/URI_scheme or
>http://www.ietf.org/rfc/rfc3986.txt .This is a "proper superset" of a
>URL. And it could encompass many other "access methods". Some that are
>implemented in browsers are things like http://some.webpage.html,
>file:///path/to/local/file, mailto:[hidden email],
>ftp://user:[hidden email]/subdir/file and lots of other
>possibilities. The RFC does not attempt to detail all possible scheme
>(the first thing in the name, before the first :). SQLite could
>implement the fetching of contents by doing something like loading a
>dynamic library at run-time which might look like: lib<scheme>.so or
>.dll Where <scheme> is the URI scheme. E.g. libftp.so or libhttp.so .
>If just used in a SELECT, SQLite would just return a string value
>equal to the value such as you might see in a browser. And if the
>contents were needed,  a function such as fetch() might be use. E.g.
>SELECT FETCH(urifield) FROM table; would get the URI for each non-NULL
>value of "urifield", inspect the first portion for the scheme, load
>the proper library and then pass the URI as a string value to that
>routine. Of course, this is getting fairly involved. And what is done
>if such a library does not exist? Error out the entire SELECT? Return
>a NULL? It gets very complicated. And I'm sure Dr. Hipp will see more
>problems and implications that I could ever imagine.
>
>
>--
>There is nothing more pleasant than traveling and meeting new people!
>Genghis Khan
>
>Maranatha! <><
>John McKown
>_______________________________________________
>sqlite-users mailing list
>[hidden email]
>http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users



_______________________________________________
sqlite-users mailing list
[hidden email]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: What if OpenDocument were built atop SQLite?

Nico Williams
On Mon, Sep 8, 2014 at 9:44 PM, Keith Medcalf <[hidden email]> wrote:
>
> Not really very difficult, and it does not require another type.  You just need a scalar function to process the URI passed to the function and return the result -- just like the readfile() (in the fileio.c extension, or included in the shell) function does for a "file-specifier" which can be opened by the platform open api.

Here's intended usage:

sqlite3> .load jq
sqlite3> -- the jq() function takes a JSON text and a jq program as
(TEXT type) arguments
sqlite3> SELECT jq(json_column, '.Title') FROM some_table WHERE
jq(json_column, '.Author.surname') LIKE 'Smith';
<titles>
sqlite3>

Now suppose you have a more complex query, with sub-queries say, so
that a result from jq() can be fed to jq() again.  Without a
pseudo-type the jq() function must always return encoded JSON texts
(as TEXT) and must always consume encoded JSON texts (as TEXT).  With
a pseudo-type the jq() function can consume either TEXT or the
pseudo-type, and it outputs the pseudo-type, which is converted to
TEXT as needed by calling the pseudo-type's encoder method.

I'm not entirely sure what you and/or John M. have in mind as for
using a URI scheme.  I assume you mean something like "encode internal
values as a URI", which is fine at first, but then you realize it
leads to unavoidable leaks.

Suppose we have a JSON C API that uses pointers (or structs with
pointers) to objects on the heap.  And then suppose that we encode
these as "URIs" as TEXT or BLOB (the latter makes encoding and
decoding easy).  But now we have a problem: if a SQL expression
concatenates/substrings these values we might not recognize the result
as a valid "URI" and we'll leak the heap object.

But with a pseudo-type there's no such risk.  If at any point a SQL
sub-expression needs to manipulate a value where one of these
pseudo-types appears then SQLite3 would call the pseudo-type's encoder
to get a JSON text (or XML, or whatever) and then apply
sub-string/concat/whatever to that.

I'd hate to have to say to users that they have to be careful with
their SQL, else they could leak parsed JSON/XML objects...  Among
other things that would be insecure.

A pseudo-type would be an optimization, really, and only an
optimization, but a very valuable one, optimizing two things in the
case of XPath and jq:

 - some XML / JSON parsing/encoding can be avoided

 - XPath / jq program compilations can be cached (like SQL statement
compilation)

For intense XML / JSON applications those two optimizations could mean a lot.

I work with one application that uses Postgres because of the hstore.
It used to support SQLite3 as well, but now it can't because SQLite3
has nothing like the hstore.  It'd be nice if SQLite3 had something
like the PG hstore, but nicer and with JSON as the text encoding.

If SQLite3 could have a nice and well-performing XML interface then
Richard Hipp's OpenDocument on SQLite3 concept would be easier to
bring to fruition.

Nico
--
_______________________________________________
sqlite-users mailing list
[hidden email]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: What if OpenDocument were built atop SQLite?

Stephan Beal-3
In reply to this post by Nico Williams
On Tue, Sep 9, 2014 at 12:45 AM, Nico Williams <[hidden email]>
wrote:

> I've played with building a JSON extension for SQLite3 using jq's
> excellent JSON C library.  The biggest "problem" with that work is

that the extension has to serialize values to JSON (and, of course,
> parse) in many cases where it could be avoided with some help from
> SQLite3.
>

FWIW:

http://fossil.wanderinghorse.net/wikis/cson/?page=cson_sqlite3


--
----- stephan beal
http://wanderinghorse.net/home/stephan/
http://gplus.to/sgbeal
"Freedom is sloppy. But since tyranny's the only guaranteed byproduct of
those who insist on a perfect world, freedom will have to do." -- Bigby Wolf
_______________________________________________
sqlite-users mailing list
[hidden email]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: What if OpenDocument were built atop SQLite?

Stephan Beal-3
On Tue, Sep 9, 2014 at 6:46 AM, Stephan Beal <[hidden email]> wrote:

> On Tue, Sep 9, 2014 at 12:45 AM, Nico Williams <[hidden email]>
> wrote:
>
>> I've played with building a JSON extension for SQLite3 using jq's
>> excellent JSON C library.  The biggest "problem" with that work is
>
> that the extension has to serialize values to JSON (and, of course,
>> parse) in many cases where it could be avoided with some help from
>> SQLite3.
>>
>
> FWIW:
>
> http://fossil.wanderinghorse.net/wikis/cson/?page=cson_sqlite3
>

BTW: that's the same code we use in Fossil for the JSON API:

https://docs.google.com/document/d/1fXViveNhDbiXgCuE7QDXQOKeFzf2qNUkBEgiUvoqFN4/view


--
----- stephan beal
http://wanderinghorse.net/home/stephan/
http://gplus.to/sgbeal
"Freedom is sloppy. But since tyranny's the only guaranteed byproduct of
those who insist on a perfect world, freedom will have to do." -- Bigby Wolf
_______________________________________________
sqlite-users mailing list
[hidden email]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: What if OpenDocument were built atop SQLite?

Nico Williams
In reply to this post by Stephan Beal-3
On Mon, Sep 8, 2014 at 11:46 PM, Stephan Beal <[hidden email]> wrote:
> FWIW:
>
> http://fossil.wanderinghorse.net/wikis/cson/?page=cson_sqlite3

Yup, I've written that sort of wrapper in Python too.  Of course, in
the preceding sub-thread I'm talking about something rather different,
but there's no denying that a JSON<->SQL wrapper is useful!

Nico
--
_______________________________________________
sqlite-users mailing list
[hidden email]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: What if OpenDocument were built atop SQLite?

justin
In reply to this post by John McKown
On 2014-09-08 13:07, John McKown wrote:
<snip>
> Open/Libre Office is Java based.
<snip>

Are you 100% about that?  When I used to be involved
with OpenOffice (years ago, prior to LibreOffice
split), the main code was C++, with Java used for
some things.  OpenOffice Base was written in Java
for example.

I *think* the Java usage came about politically
because Sun (back in the day) decided Java _going_
to be the language of choice.  Regardless of developer
sentiment.  So they went about developing / bolting on
all of the new pieces with it.

That's my dodgy memory of things anyway.  LibreOffice
might have changed things, etc. ;)

Regards and best wishes,

Justin Clift
_______________________________________________
sqlite-users mailing list
[hidden email]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: What if OpenDocument were built atop SQLite?

John McKown
Well, I did a git clone to fetch the LibreOffice source. It appears to
be a mixture of Java, C, and C++. Just some stats:

$find . -name '*.c' | wc
    108     108    3908
~/source-oem/libreoffice-core$find . -name '*.cpp' | wc
     26      26    1360
~/source-oem/libreoffice-core$find . -name '*.java' | wc
   3329    3329  206921
~/source-oem/libreoffice-core$find . -name '*.cxx' | wc
   9562    9562  405662

And, for "fun", I ran: for i in c cpp cxx java;do echo "$i";wc $(find
. -name "*.${i}");done |& tee mckown
to get the number of lines of code in each of those.
c == 45,322 lines
cpp == 4,600 lines
cxx == 5,162,525
java == 616,578 lines

So C++ "wins" by about a 3:1 ratio of number of files and 10:1 ratio
in terms of lines of code over Java.

On Tue, Sep 9, 2014 at 6:58 PM,  <[hidden email]> wrote:

> On 2014-09-08 13:07, John McKown wrote:
> <snip>
>>
>> Open/Libre Office is Java based.
>
> <snip>
>
> Are you 100% about that?  When I used to be involved
> with OpenOffice (years ago, prior to LibreOffice
> split), the main code was C++, with Java used for
> some things.  OpenOffice Base was written in Java
> for example.
>
> I *think* the Java usage came about politically
> because Sun (back in the day) decided Java _going_
> to be the language of choice.  Regardless of developer
> sentiment.  So they went about developing / bolting on
> all of the new pieces with it.
>
> That's my dodgy memory of things anyway.  LibreOffice
> might have changed things, etc. ;)
>
> Regards and best wishes,
>
> Justin Clift
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users



--
There is nothing more pleasant than traveling and meeting new people!
Genghis Khan

Maranatha! <><
John McKown
_______________________________________________
sqlite-users mailing list
[hidden email]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: What if OpenDocument were built atop SQLite?

Gabor Grothendieck
On Wed, Sep 10, 2014 at 8:36 AM, John McKown
<[hidden email]> wrote:

> Well, I did a git clone to fetch the LibreOffice source. It appears to
> be a mixture of Java, C, and C++. Just some stats:
>
> $find . -name '*.c' | wc
>     108     108    3908
> ~/source-oem/libreoffice-core$find . -name '*.cpp' | wc
>      26      26    1360
> ~/source-oem/libreoffice-core$find . -name '*.java' | wc
>    3329    3329  206921
> ~/source-oem/libreoffice-core$find . -name '*.cxx' | wc
>    9562    9562  405662
>
> And, for "fun", I ran: for i in c cpp cxx java;do echo "$i";wc $(find
> . -name "*.${i}");done |& tee mckown
> to get the number of lines of code in each of those.
> c == 45,322 lines
> cpp == 4,600 lines
> cxx == 5,162,525
> java == 616,578 lines
>
> So C++ "wins" by about a 3:1 ratio of number of files and 10:1 ratio
> in terms of lines of code over Java.
>

There is also a breakdown of libre office code here with graphs and a table:
https://www.openhub.net/p/libreoffice/analyses/latest/languages_summary

The corresponding page for sqlite is:
https://www.openhub.net/p/sqlite/analyses/latest/languages_summary
_______________________________________________
sqlite-users mailing list
[hidden email]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: What if OpenDocument were built atop SQLite?

Richard Hipp-3
On Wed, Sep 10, 2014 at 9:02 AM, Gabor Grothendieck <[hidden email]
> wrote:

>
> There is also a breakdown of libre office code here with graphs and a
> table:
> https://www.openhub.net/p/libreoffice/analyses/latest/languages_summary
>
> The corresponding page for sqlite is:
> https://www.openhub.net/p/sqlite/analyses/latest/languages_summary
>

The veracity of these graphs is questionable.  I don't know the LibreOffice
code base, but I am quite familiar with SQLite, and the LOC values for
SQLite are way off.

According to the charts above, SQLite as 15K lines of TCL code.  I count
319K lines - more than 20x as much.  On the other hand, they list 27K lines
of shell script whereas if you exclude authconf the actual amount is closer
to 270 lines.  The charts show 8544 lines of C++ whereas the correct amount
is 0.


--
D. Richard Hipp
[hidden email]
_______________________________________________
sqlite-users mailing list
[hidden email]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users