Possible feature request 'Raw Row' representation

classic Classic list List threaded Threaded
2 messages Options
w b
Reply | Threaded
Open this post in threaded view
|

Possible feature request 'Raw Row' representation

w b
   
Hi,

I was wondering if this could be a useful addition to the
SQLITE API set. I'm not sure if this possible to create
so figured I'd post it here for further discussion
rather than blindly putting it in as an enhancment request.

Just for some background I have an application that runs
on multiple machines (and different platforms) and periodicially
each one sends various statistics about its health to the
other applications. The stats are computed and stored in a
single table for historical purposes, but what gets sent
out to the other machines is the latest row from that table.

At the moment when I pull back the row. I package it
up in to a network byte order representation and send
it on its merry way. The recv's of course unpack the
message and then refresh their local information
with the information sent, again in a SQLITE DB
having the same table structure as what it originated from.

So I was thinking (Dangerous with out too much coffee!)
if the DB format is platform neutral is there a way to
obtain a raw image of that row from SQLITE rather than
performing the sqlite3_column_XXX for each field returned.

If a raw row representation was available say via the likes of an API

const void *sqlite3_get_rawrow(sqlite3_stmt*)

I could then skip the sqlite3_column_xxx operations as well as some
of the network byte order conversion routines and send that to the
various receivers. (The assumption here is of course that the
byte array returned is in a platform neutral representation)

On the receiving side there could be a similar set of API's to
insert the raw row into the corresponding table

int sqlite3_insert_rawrow(sqlite3*, const char* tablename, void
*rawrow);

Or alternatively if the row was just to be discarded after reading
there could be something like the following to allow you to
still perform the sqlite3_column_XXX requests on it

sqlite3_stmt* sqlite3_set_rawrow_result(sqlite3*, void *rawrow)


While I have an application that could benefit from this, I guess it
could
be employed in other situations as well such as forming some form
of replication deamon between two DB's, or creating a binary table
dump that could be more compact than a text export  


Well thats about it, let the responses begin

Wayne
 
Reply | Threaded
Open this post in threaded view
|

Re: Possible feature request 'Raw Row' representation

Andrew Piskorski
On Thu, Feb 16, 2006 at 06:09:15PM -0800, w b wrote:

> At the moment when I pull back the row. I package it up in to a
> network byte order representation and send it on its merry way. The
> recv's of course unpack the message and then refresh their local
> information with the information sent, again in a SQLITE DB having
> the same table structure as what it originated from.
>
> So I was thinking (Dangerous with out too much coffee!)  if the DB
> format is platform neutral is there a way to obtain a raw image of
> that row from SQLITE rather than performing the sqlite3_column_XXX
> for each field returned.

That might be an interesting or useful technique for other uses as
well, say if you were trying to make clusters of SQLite engines act as
one consistent database.  Your "raw row" idea sounds at least
superficially related to what the old Postgres-R project did for
multi-master replication.

In Postgres-R, basically each node would process the SQL for a
transaction, then ship a much lower level representation of that
transaction to the other nodes, using one of the academic group
communicatins toolkits, either Spread or (I think) Ensemble.  Since
for many transactions applying this lower level representation was
much more efficient than processing the original SQL, this gave some
scalability.

Bettina Kemme's paper from 2000, "Don't be lazy, be consistent:
Postgres-R, a new way to implement Database Replication." describes
it:

  http://openacs.org/forums/message-view?message_id=128917
  http://www.informatik.uni-trier.de/~ley/db/conf/vldb/KemmeA00.html

Postgres-R was research-ware based on an ancient 6.x version of
PostgreSQL (from before MVCC!), and AFAIK no one ever drummed up
enough interest and manpower to actually finish forward porting it to
a more recent PostgreSQL version.  But it seemed interesting.

--
Andrew Piskorski <[hidden email]>
http://www.piskorski.com/