Shared Memory Question

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Shared Memory Question

Drew, Stephen
Hello all,
 
An interesting use of the in-memory SQLite database would be to use it
in shared memory and have the data shared between applications but
stored once.
 
The background to this is I am making local replicated copies of a main
database, and as they are transient (they are read-only copies of the
main database), there is no real need to have them on disk, or at least
not permanently.
 
To get around locking issues, obviously the whole database would need to
be wrapped in a mutex (simple, slow approach), but what I was wondering
is:
 
Presumably, the in-memory database is in contiguous memory, as I guess
the format is the same as the disk-based database format.  Would it be a
lot of work to be able to construct an in-memory database with a chunk
of memory (allocated by the user), and a size?  The database could then
be created in shared memory, and a SQLITE_FULL error returned if the
database attempted to grow outside this size limit.  The shared memory
segment could be flushed to disk independently and when the application
decides.
 
Is this a reasonable request and would this be useful to anyone else?
Or is there something already present that I could use?
 
Many thanks,
Steve
 
Reply | Threaded
Open this post in threaded view
|

RE: Shared Memory Question

Drew, Stephen
Sorry, just noticed this is very similar Mateus' email from earlier, so
will wait for any responses to that...

-----Original Message-----
From: Drew, Stephen
Sent: 15 February 2006 15:33
To: [hidden email]
Subject: [sqlite] Shared Memory Question

Hello all,
 
An interesting use of the in-memory SQLite database would be to use it
in shared memory and have the data shared between applications but
stored once.
 
The background to this is I am making local replicated copies of a main
database, and as they are transient (they are read-only copies of the
main database), there is no real need to have them on disk, or at least
not permanently.
 
To get around locking issues, obviously the whole database would need to
be wrapped in a mutex (simple, slow approach), but what I was wondering
is:
 
Presumably, the in-memory database is in contiguous memory, as I guess
the format is the same as the disk-based database format.  Would it be a
lot of work to be able to construct an in-memory database with a chunk
of memory (allocated by the user), and a size?  The database could then
be created in shared memory, and a SQLITE_FULL error returned if the
database attempted to grow outside this size limit.  The shared memory
segment could be flushed to disk independently and when the application
decides.
 
Is this a reasonable request and would this be useful to anyone else?
Or is there something already present that I could use?
 
Many thanks,
Steve
 


Reply | Threaded
Open this post in threaded view
|

Re: Shared Memory Question

D. Richard Hipp
In reply to this post by Drew, Stephen
"Drew, Stephen" <[hidden email]> wrote:
>  
> Presumably, the in-memory database is in contiguous memory....

Sorry, that is not the case.

> as I guess
> the format is the same as the disk-based database format.  Would it be a
> lot of work to be able to construct an in-memory database with a chunk
> of memory (allocated by the user), and a size?  The database could then
> be created in shared memory, and a SQLITE_FULL error returned if the
> database attempted to grow outside this size limit.  The shared memory
> segment could be flushed to disk independently and when the application
> decides.
>  
> Is this a reasonable request and would this be useful to anyone else?
> Or is there something already present that I could use?
>  

If you set

   PRAGMA synchronous=OFF

then SQLite will still think it is writing to disk, but the operating
system will not actually flush those writes to disk until it is ready.
Most of the data will be held in the operating systems disk cache.
This makes things go a lot faster (which is presumably what you are
trying to accomplish) but means that the database will likely be
corrupted if you take an unexpected power loss or operating system
crash.  (Application crashes will *not* corrupt the database when
synchronous=OFF - only a power failure or operating system crash can
do that.)

--
D. Richard Hipp   <[hidden email]>

Reply | Threaded
Open this post in threaded view
|

Re: Shared Memory Question

Kervin L. Pierre
In reply to this post by Drew, Stephen
Hello,

We are currently looking at this very issue
ourselves.

I just put in an enhancement request...
http://www.sqlite.org/cvstrac/tktview?tn=1679
For allowing the runtime replacement of memory
functions by using function pointers instead
of C mallocs.  We could then replace SQLite's
allocators with a memory-pool based allocator
for speed.  The application's memory-pool would
be allocated on shared-memory for debugging
in-memory database.  Findly SQLiteBrowser
[ http://sqlitebrowser.sourceforge.net/ ] would
be modified to look to the shared-memory segment
for the in-memory database ( that should be easy
)

Best Regards,
Kervin
Reply | Threaded
Open this post in threaded view
|

Re: Shared Memory Question

Glenn McAllister
In reply to this post by Drew, Stephen
This is likely a naive response, but on Linux have you thought using
/dev/shm?  It's a tmpfs ramdisk that is needed by POSIX shared memory
calls shm_open and shm_unlink in glibc 2.2 and above.  It grows and
shrinks as required and uses almost no memory if it's never populated
with files.

As a simple test I created /dev/shm/test.d using sqlite3, created a
simple table and populated it with a couple of rows of data.  I
connected to the database from another sqlite instance, and I was able
to read the data just fine.  After closing down both instances the test
database was still there (no surprise, it's a filesystem after all).

Dunno if that helps any.

Glenn McAllister
SOMA Networks, Inc.

Drew, Stephen wrote:
> Hello all,
>  
> An interesting use of the in-memory SQLite database would be to use it
> in shared memory and have the data shared between applications but
> stored once.
Reply | Threaded
Open this post in threaded view
|

RE: Shared Memory Question

Drew, Stephen
In reply to this post by Drew, Stephen
Glenn,

I certainly would, but sadly this is for Win32 too.

Cheers,
Steve

-----Original Message-----
From: Glenn McAllister [mailto:[hidden email]]
Sent: 15 February 2006 16:45
To: [hidden email]
Subject: Re: [sqlite] Shared Memory Question

This is likely a naive response, but on Linux have you thought using
/dev/shm?  It's a tmpfs ramdisk that is needed by POSIX shared memory
calls shm_open and shm_unlink in glibc 2.2 and above.  It grows and
shrinks as required and uses almost no memory if it's never populated
with files.

As a simple test I created /dev/shm/test.d using sqlite3, created a
simple table and populated it with a couple of rows of data.  I
connected to the database from another sqlite instance, and I was able
to read the data just fine.  After closing down both instances the test
database was still there (no surprise, it's a filesystem after all).

Dunno if that helps any.

Glenn McAllister
SOMA Networks, Inc.

Drew, Stephen wrote:
> Hello all,
>  
> An interesting use of the in-memory SQLite database would be to use it

> in shared memory and have the data shared between applications but
> stored once.


Reply | Threaded
Open this post in threaded view
|

Re: Shared Memory Question

Mateus Cordeiro Inssa
In reply to this post by Drew, Stephen
> This is likely a naive response, but on Linux have you thought
>using /dev/shm? It's a tmpfs ramdisk that is needed by POSIX shared memory
>calls shm_open and shm_unlink in glibc 2.2 and above. It grows and shrinks as
>required and uses almost no memory if it's never populated with files.    
>
>As a simple test I created /dev/shm/test.d using sqlite3, created a simple
>table and populated it with a couple of rows of data. I connected to the
>database from another sqlite instance, and I was able to read the data just
>fine. After closing down both instances the test database was still there (no
>surprise, it's a filesystem after all).    
>
> Dunno if that helps any.
>
> Glenn McAllister
> SOMA Networks, Inc.

  Hmm, great idea! I forgot that .... This solves the problem with little
databases, really.

  But as I said in another e-mail, I need to use shared memories greater than
5GB .... :-) The problem with lots of RAM is the work the kernel has to do to
control all this memory .... Think: each memory page (in linux) is 4kbytes.
How many pages are necessary to map 8GB of memory ? So, using shared memory
created with the flag SHM_HUGETLB (HugeTLB file system support), each memory
page has the size of 4M (with kernels compiled with 4GB option) and 2M (with
kernels compiled with 64GB option). Using pages with 2M or 4M saves a lot of
kernel work.

  The tmpfs doesn't use this strategy, having the same performance of
malloc'ed memory (but the locks will work and it's possible to share the
database with other processes :-)

  []'s

Mateus Inssa