Feature request

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Feature request

Cariotoglou Mike
Hi all. I would like to propose an enchancement in sqlite. I am not sure whether the issue
has come up before, if it has, and has been shot down, pls let me know.

The concept of in-memory database would be greatly enhanced if there was a mechanism to
serialize the internal cache to/from a stream (not necessarily a disk stream). In-memory
databases are extremely useful for a number of applications, but ar a PITA to initalize/persist.

one might argue why such a mechanism would be useful. If I need a persistent database, you could
say, then I would make it a disk-based one. Not so. Consider the case where you would like to
store the contents of an entire "database" as part of another storage structure.
consider the case where an entire sqlite database was a field in another RDBMS. consider
OO databases. Consider case tool implementation. The applications would be endless.

I took a look at the ":memory:" implementation, and it seems to me that this enchancement
would be trivial, consisting of maybe 10-20 c code lines, for somebody who understands the
internal structures involved. unfortunately, I do not do C, so I cannot do this myself.
OTOH, given the extreme usefuleness of what I propose (IMHO), could you consider this as a
permament addition in sqlite?

the design I have in mind would be something like this:

int sqlite3_loadMemDb(sqlite3 * db,reader)
int sqlite3_saveMemDb(sqlite3 * db,writer)

where "reader" and "writer" are function pointers with a signature like :
 int reader(void mem, int size)

(excuse my attempt at C syntax, this is meant only as a tip)

I suspect that the "load" function might need to know the number of pages beforehand,
so some kind of overloaded definiton of "reader" would be required, that would return this
information. if we agree that the idea has merit, the details can be worked out easily.

what does the community, and especially DRH, think about this ?


Reply | Threaded
Open this post in threaded view
|

Re: Feature request

Will Leshner-2

On Nov 30, 2005, at 4:18 PM, Cariotoglou Mike wrote:

> The concept of in-memory database would be greatly enhanced if  
> there was a mechanism to
> serialize the internal cache to/from a stream (not necessarily a  
> disk stream). In-memory
> databases are extremely useful for a number of applications, but ar  
> a PITA to initalize/persist.

With the very exciting virtual OsFile API that is being developed,  
I'm wondering if we couldn't serialize a SQLite database ourselves. I  
haven't looked at the entire API, so maybe this isn't possible, but  
if you can create multiple "subclasses" of OsFile in the same  
application, then you could have an on-disk subclass for your file  
system, and an in-memory one as well, and theoretically SQLite  
wouldn't know the difference. Then you could attach one to another  
and copy tables back and forth.
Reply | Threaded
Open this post in threaded view
|

Re: Feature request

i-love-spam
In reply to this post by Cariotoglou Mike
Cariotoglou Mike wrote:


Personally, I also was looking how to make sqlite database be in-memory
only (or, better, in shared memory). Then I just created ram based
filesystem (on freebsd) and my database file on this filesystem. It
would be nice if sqlite supported memory only functionality without
unnecesary memory cashes etc in this case.



> Hi all. I would like to propose an enchancement in sqlite. I am not sure whether the issue
> has come up before, if it has, and has been shot down, pls let me know.
>
> The concept of in-memory database would be greatly enhanced if there was a mechanism to
> serialize the internal cache to/from a stream (not necessarily a disk stream). In-memory
> databases are extremely useful for a number of applications, but ar a PITA to initalize/persist.
>
> one might argue why such a mechanism would be useful. If I need a persistent database, you could
> say, then I would make it a disk-based one. Not so. Consider the case where you would like to
> store the contents of an entire "database" as part of another storage structure.
> consider the case where an entire sqlite database was a field in another RDBMS. consider
> OO databases. Consider case tool implementation. The applications would be endless.
>
> I took a look at the ":memory:" implementation, and it seems to me that this enchancement
> would be trivial, consisting of maybe 10-20 c code lines, for somebody who understands the
> internal structures involved. unfortunately, I do not do C, so I cannot do this myself.
> OTOH, given the extreme usefuleness of what I propose (IMHO), could you consider this as a
> permament addition in sqlite?
>
> the design I have in mind would be something like this:
>
> int sqlite3_loadMemDb(sqlite3 * db,reader)
> int sqlite3_saveMemDb(sqlite3 * db,writer)
>
> where "reader" and "writer" are function pointers with a signature like :
>  int reader(void mem, int size)
>
> (excuse my attempt at C syntax, this is meant only as a tip)
>
> I suspect that the "load" function might need to know the number of pages beforehand,
> so some kind of overloaded definiton of "reader" would be required, that would return this
> information. if we agree that the idea has merit, the details can be worked out easily.
>
> what does the community, and especially DRH, think about this ?
>
>

Reply | Threaded
Open this post in threaded view
|

RE: Feature request

Cariotoglou Mike
In reply to this post by Cariotoglou Mike
I did not know about this one. where can I get info ?

________________________________

From: Will Leshner [mailto:[hidden email]]
Sent: Thu 01-Dec-05 2:46 AM
To: [hidden email]
Subject: Re: [sqlite] Feature request





With the very exciting virtual OsFile API that is being developed,
I'm wondering if we couldn't serialize a SQLite database ourselves. I
haven't looked at the entire API, so maybe this isn't possible, but
if you can create multiple "subclasses" of OsFile in the same
application, then you could have an on-disk subclass for your file
system, and an in-memory one as well, and theoretically SQLite
wouldn't know the difference. Then you could attach one to another
and copy tables back and forth.



Reply | Threaded
Open this post in threaded view
|

RE: Feature request

Drew, Stephen
In reply to this post by Cariotoglou Mike
I would be in favour of this, specifically to allow shared memory
database access.

-----Original Message-----
From: Cariotoglou Mike [mailto:[hidden email]]
Sent: 01 December 2005 00:18
To: [hidden email]
Subject: [sqlite] Feature request

Hi all. I would like to propose an enchancement in sqlite. I am not sure
whether the issue has come up before, if it has, and has been shot down,
pls let me know.

The concept of in-memory database would be greatly enhanced if there was
a mechanism to serialize the internal cache to/from a stream (not
necessarily a disk stream). In-memory databases are extremely useful for
a number of applications, but ar a PITA to initalize/persist.

one might argue why such a mechanism would be useful. If I need a
persistent database, you could say, then I would make it a disk-based
one. Not so. Consider the case where you would like to store the
contents of an entire "database" as part of another storage structure.
consider the case where an entire sqlite database was a field in another
RDBMS. consider OO databases. Consider case tool implementation. The
applications would be endless.

I took a look at the ":memory:" implementation, and it seems to me that
this enchancement would be trivial, consisting of maybe 10-20 c code
lines, for somebody who understands the internal structures involved.
unfortunately, I do not do C, so I cannot do this myself.
OTOH, given the extreme usefuleness of what I propose (IMHO), could you
consider this as a permament addition in sqlite?

the design I have in mind would be something like this:

int sqlite3_loadMemDb(sqlite3 * db,reader) int sqlite3_saveMemDb(sqlite3
* db,writer)

where "reader" and "writer" are function pointers with a signature like
:
 int reader(void mem, int size)

(excuse my attempt at C syntax, this is meant only as a tip)

I suspect that the "load" function might need to know the number of
pages beforehand, so some kind of overloaded definiton of "reader" would
be required, that would return this information. if we agree that the
idea has merit, the details can be worked out easily.

what does the community, and especially DRH, think about this ?




Reply | Threaded
Open this post in threaded view
|

RE: Feature request

Cariotoglou Mike
In reply to this post by Cariotoglou Mike
I took a look at the code. My perception is that, while the new design
*can* imnplement what I want, it cannot do so without writing code
at the C level. since I am using sqlite via the dll "wrapper", I do not
have this option, I would need an "api" level capability for this,
which is not the same thing as the new virtual OsFile.

> -----Original Message-----
> From: Will Leshner [mailto:[hidden email]]
> Sent: Thursday, December 01, 2005 2:46 AM
> To: [hidden email]
> Subject: Re: [sqlite] Feature request
>
>
> On Nov 30, 2005, at 4:18 PM, Cariotoglou Mike wrote:
>
> > The concept of in-memory database would be greatly enhanced
> if there
> > was a mechanism to serialize the internal cache to/from a
> stream (not
> > necessarily a disk stream). In-memory databases are
> extremely useful
> > for a number of applications, but ar a PITA to initalize/persist.
>
> With the very exciting virtual OsFile API that is being
> developed, I'm wondering if we couldn't serialize a SQLite
> database ourselves. I haven't looked at the entire API, so
> maybe this isn't possible, but if you can create multiple
> "subclasses" of OsFile in the same application, then you
> could have an on-disk subclass for your file system, and an
> in-memory one as well, and theoretically SQLite wouldn't know
> the difference. Then you could attach one to another and copy
> tables back and forth.
>
>
>