SQLite as a IPC mechanism

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

SQLite as a IPC mechanism

Joshua D. Boyd
I have a system that currently consists of 2 C programs and 3 python
programs.  Currently the python programs transfer data between
themselves via pickles.  The C programs transfer data between themselves
via streaming structs, and the C programs talk to one of the python
programs via a fairly ugly text over socket method.  All of the programs
are threaded.

Of the data being communicated, some of it must also be saved to disk,
and other pieces go away after a reset.  All told there is only about 4
k of stuff saved.

I am wondering about using SQLite to communicate between the programs.
I'd use two databases.  One on a flash disk for the data that needs to
be saved, and the other database would somehow be in a ram disk.  Each
Db would have 1 table, and the fields would be key, type, val.  Most
fields would only be written to by one or two sources, but would be read
from by nearly all processes.

Is this a stupid use of SQLite?  I can't quite seem to find anyone using
it like this.  I am a little concerned about page locking as opposed to
row locking, but I think I can work around that.

-----------------------------------------------------------------------------
To unsubscribe, send email to [hidden email]
-----------------------------------------------------------------------------

Reply | Threaded
Open this post in threaded view
|

Re: SQLite as a IPC mechanism

Clay Dowling
Seems like it would work, but maybe overkill.  What's stopping you from
working out a fairly direct protocol to exchange data with?  Sending key,
type, val for all of your IPC seems reasonable.  A mem-mapped file, a
local socket or a network socket seem reasonable, depending on the
structure of the system.

Clay


Joshua D. Boyd wrote:

> I have a system that currently consists of 2 C programs and 3 python
> programs.  Currently the python programs transfer data between
> themselves via pickles.  The C programs transfer data between themselves
> via streaming structs, and the C programs talk to one of the python
> programs via a fairly ugly text over socket method.  All of the programs
> are threaded.
>
> Of the data being communicated, some of it must also be saved to disk,
> and other pieces go away after a reset.  All told there is only about 4
> k of stuff saved.
>
> I am wondering about using SQLite to communicate between the programs.
> I'd use two databases.  One on a flash disk for the data that needs to
> be saved, and the other database would somehow be in a ram disk.  Each
> Db would have 1 table, and the fields would be key, type, val.  Most
> fields would only be written to by one or two sources, but would be read
> from by nearly all processes.
>
> Is this a stupid use of SQLite?  I can't quite seem to find anyone using
> it like this.  I am a little concerned about page locking as opposed to
> row locking, but I think I can work around that.
>
> -----------------------------------------------------------------------------
> To unsubscribe, send email to [hidden email]
> -----------------------------------------------------------------------------
>
>


--
Lazarus Registration
http://www.lazarusid.com/registration.shtml


-----------------------------------------------------------------------------
To unsubscribe, send email to [hidden email]
-----------------------------------------------------------------------------

Reply | Threaded
Open this post in threaded view
|

Re: SQLite as a IPC mechanism

ken-33
In reply to this post by Joshua D. Boyd
Joshua,
I don't think sqlite was designed to be used that way.

It would work very well for your persistent data, that is written to disk.

However, not so well using it as an IPC. The database is locked, not the table, row or even page. The entire DB is locked when you write (insert, update or delete) data.

IPC, is best done as either Message Queues depending upon your data volumes. Shared memory with appropriate semaphores/ Mutexes.
Or even as one person posted Mmap files. You'll stilll need a mutex or a semaphore to write.


Ken


"Joshua D. Boyd" <[hidden email]> wrote: I have a system that currently consists of 2 C programs and 3 python
programs.  Currently the python programs transfer data between
themselves via pickles.  The C programs transfer data between themselves
via streaming structs, and the C programs talk to one of the python
programs via a fairly ugly text over socket method.  All of the programs
are threaded.

Of the data being communicated, some of it must also be saved to disk,
and other pieces go away after a reset.  All told there is only about 4
k of stuff saved.

I am wondering about using SQLite to communicate between the programs.
I'd use two databases.  One on a flash disk for the data that needs to
be saved, and the other database would somehow be in a ram disk.  Each
Db would have 1 table, and the fields would be key, type, val.  Most
fields would only be written to by one or two sources, but would be read
from by nearly all processes.

Is this a stupid use of SQLite?  I can't quite seem to find anyone using
it like this.  I am a little concerned about page locking as opposed to
row locking, but I think I can work around that.

-----------------------------------------------------------------------------
To unsubscribe, send email to [hidden email]
-----------------------------------------------------------------------------


Reply | Threaded
Open this post in threaded view
|

Re: SQLite as a IPC mechanism

Roger Binns
In reply to this post by Joshua D. Boyd
Joshua D. Boyd wrote:
> I have a system that currently consists of 2 C programs and 3 python
> programs.  Currently the python programs transfer data between
> themselves via pickles.  The C programs transfer data between themselves
> via streaming structs, and the C programs talk to one of the python
> programs via a fairly ugly text over socket method.  All of the programs
> are threaded.

Since you posted using a Linux mailer I'd assume your system is Linux
based.  I think you'll find that dbus is an excellent solution to your
requirements and it works just fine with C and python based programs.

http://en.wikipedia.org/wiki/D-Bus
http://www.freedesktop.org/wiki/Software/dbus

D-Bus deals with threading just fine.  It also has a nice notification
mechanism named 'signals'.

Roger

-----------------------------------------------------------------------------
To unsubscribe, send email to [hidden email]
-----------------------------------------------------------------------------