Custom serialization/deserialization of in-memory DB

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Custom serialization/deserialization of in-memory DB

Zoltan Demeter
Hello there,

I am working on a small plugin for a bigger application. I am bound by
the in-house app framework, for example I cannot create arbitrary
files or use anything OS-specific directly.

I am using SQLite with an in-memory DB to obtain sorted, filtered
lists of items. I am happy with the library. The only problem is, the
amount of data can be so huge, that it takes a considerable amount of
time to calculate some of the fields and build up the DB.

Thus, I would like to save my tables, but the "normal" backup API
wants to use a file. Like I wrote, I cannot use an arbitrary file, my
load/save procedure must be integrated into the app framework so that
all data will be packed into the project data file(s).

Basically, if I could obtain a byte array, I could save it to disk.
Load and save should work across bitness and OS barriers ... eg.
between 32-bit Windows and 64-bit Mac.

Or if I could tell SQLite to use custom file I/O routines ... I could
mock fwrite() to just write to memory and then grab the data.

A "simple" table dump, without indices etc. would be already be great.

Any ideas please?
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Custom serialization/deserialization of in-memory DB

Shawn Wagner
https://www.sqlite.org/c3ref/serialize.html might help.

On Mon, Oct 22, 2018, 11:31 AM Zoltan Demeter <[hidden email]> wrote:

> Hello there,
>
> I am working on a small plugin for a bigger application. I am bound by
> the in-house app framework, for example I cannot create arbitrary
> files or use anything OS-specific directly.
>
> I am using SQLite with an in-memory DB to obtain sorted, filtered
> lists of items. I am happy with the library. The only problem is, the
> amount of data can be so huge, that it takes a considerable amount of
> time to calculate some of the fields and build up the DB.
>
> Thus, I would like to save my tables, but the "normal" backup API
> wants to use a file. Like I wrote, I cannot use an arbitrary file, my
> load/save procedure must be integrated into the app framework so that
> all data will be packed into the project data file(s).
>
> Basically, if I could obtain a byte array, I could save it to disk.
> Load and save should work across bitness and OS barriers ... eg.
> between 32-bit Windows and 64-bit Mac.
>
> Or if I could tell SQLite to use custom file I/O routines ... I could
> mock fwrite() to just write to memory and then grab the data.
>
> A "simple" table dump, without indices etc. would be already be great.
>
> Any ideas please?
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Custom serialization/deserialization of in-memory DB

Simon Slavin-3
In reply to this post by Zoltan Demeter
On 22 Oct 2018, at 7:30pm, Zoltan Demeter <[hidden email]> wrote:

> Or if I could tell SQLite to use custom file I/O routines ... I could mock fwrite() to just write to memory and then grab the data.

You can tell indeed SQLite to use custom file I/O routines ... by writing your own virtual filesystem (VFS):

<https://sqlite.org/c3ref/vfs.html>

It's not for the faint-hearted, but it does allow you to do anything you want instead of using your operating system's normal file system.  So you could write to reserved memory, or write to a custom-formatted Flash drive, or transmit it via your device's satellite feed.

Several example VFS implementations are included in the /full/ download of SQLite (not the amalgamation download).  You might want to look at test_onefile.c as documented here:

<https://sqlite.org/vfs.html>

Simon.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users