kooky thought: a vm-only build (for embedded). feasible?

[dave]

I had a stray thought, and wanted to ask if it's been thunk before,and if so
what is the thinking?  Or just for commentary.
 
I have been building a system, part of which uses sqlite and virtual tables.
This is working great in a desktop/mobile environment.  However, eventually
one day, I will want to migrate aspects of the product to deeply embedded
systems (e.g. something like an STM32F4 class chip), and am thinking about
size -- both code and RAM.  I know about the various compile switches that
can turn off various features, but I wonder if I can really strip it down
further by eliminating parsing, query planning, etc, altogether, and only
support the virtual machine.  I do need virtual tables, though.  In my
particular use-case, I only need read access -- no create or update.  The
thinking being that I can build queries offline and compile them into the
p-code (or whatever it's called), and either burn those well know queries
into flash, or perhaps send them down the wire as needed.  Then of course
(maybe even more critically), can I control ram usage in a deterministic way
such that it will still work on memory-constrained devices (e.g. having a
total of 128 KiB max for the whole system).
 
Anway, has this been discussed before?  Or is it a fool's errand?
 
Cheers!
 
-dave
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[Simon Slavin-3]

On 15 Apr 2018, at 7:54pm, dave <[hidden email]> wrote:

> I wonder if I can really strip it down
> further by eliminating parsing, query planning, etc, altogether, and only
> support the virtual machine.

I wonder what you would find if you looked through the data structure of sqlite3_stmt.  Presumably the compilation process would convert the SQL text into bytecode and the bytecode would be stored in the statement.

Once you have seen the bytecode from your desired SQL, it might be possible to write a C function which accepts a pointer and a length and creates a statement with a copy of that chunk of memory as the bytecode and everything else set up the way it is in a newly-created statement.

Or something like that.

Simon.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[dave]

That was more-or-less my thinking.

Mostly, my inquiry is to solicit any advice or wisdom-of-the-ages, or even
advice against it.  After having sent that, I suspect that this would more
likely wind up being something I'm on my own in doing the hands-on work, but
I still welcome any advice on how to approach the surgery.

-dave

> -----Original Message-----
> From: sqlite-users
> [mailto:[hidden email]] On
> Behalf Of Simon Slavin
> Sent: Sunday, April 15, 2018 2:06 PM
> To: SQLite mailing list
> Subject: Re: [sqlite] kooky thought: a vm-only build (for
> embedded).feasible?
>
>
>
>
> On 15 Apr 2018, at 7:54pm, dave <[hidden email]> wrote:
>
> > I wonder if I can really strip it down
> > further by eliminating parsing, query planning, etc,
> altogether, and only
> > support the virtual machine.
>
> I wonder what you would find if you looked through the data
> structure of sqlite3_stmt.  Presumably the compilation
> process would convert the SQL text into bytecode and the
> bytecode would be stored in the statement.
>
> Once you have seen the bytecode from your desired SQL, it
> might be possible to write a C function which accepts a
> pointer and a length and creates a statement with a copy of
> that chunk of memory as the bytecode and everything else set
> up the way it is in a newly-created statement.
>
> Or something like that.
>
> Simon.
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[Richard Hipp-3]

On 4/15/18, dave <[hidden email]> wrote:

> I had a stray thought, and wanted to ask if it's been thunk before,and if so
> what is the thinking?  Or just for commentary.
>
> I have been building a system, part of which uses sqlite and virtual tables.
> This is working great in a desktop/mobile environment.  However, eventually
> one day, I will want to migrate aspects of the product to deeply embedded
> systems (e.g. something like an STM32F4 class chip), and am thinking about
> size -- both code and RAM.  I know about the various compile switches that
> can turn off various features, but I wonder if I can really strip it down
> further by eliminating parsing, query planning, etc, altogether, and only
> support the virtual machine.  I do need virtual tables, though.  In my
> particular use-case, I only need read access -- no create or update.  The
> thinking being that I can build queries offline and compile them into the
> p-code (or whatever it's called), and either burn those well know queries
> into flash, or perhaps send them down the wire as needed.  Then of course
> (maybe even more critically), can I control ram usage in a deterministic way
> such that it will still work on memory-constrained devices (e.g. having a
> total of 128 KiB max for the whole system).
>
> Anway, has this been discussed before?  Or is it a fool's errand?

We did this once, back in 2005, for a startup company in Boston.  It
was called "SSE".  Unfortunately, we didn't continue to support it.  I
went looking for the source code and could not find it.

The database was to run on a smart-card with limited RAM.  All of the
prepared statements were generated on a workstation, then serialized
and stored in a special table in the database file.  The application
would then use a special API that would deserialize a prepared
statement (identified by a well-known integer) then bind parameters
and run it.

So much has changed in the SQLite bytecode engine since then that
there is basically zero chance that SSE would still run today, even if
I could find the source code.

--
D. Richard Hipp
[hidden email]
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[Hick Gunter]

There have been some inquries on the list into executing pre-generated bytecode. If you have a fixed schema you may be able to create a valid sqlite3_stmt pointer from bytecode stored somewhere.

-----Ursprüngliche Nachricht-----
Von: sqlite-users [mailto:[hidden email]] Im Auftrag von dave
Gesendet: Sonntag, 15. April 2018 20:55
An: [hidden email]
Betreff: [EXTERNAL] [sqlite] kooky thought: a vm-only build (for embedded). feasible?

I had a stray thought, and wanted to ask if it's been thunk before,and if so what is the thinking?  Or just for commentary.

I have been building a system, part of which uses sqlite and virtual tables.
This is working great in a desktop/mobile environment.  However, eventually one day, I will want to migrate aspects of the product to deeply embedded systems (e.g. something like an STM32F4 class chip), and am thinking about size -- both code and RAM.  I know about the various compile switches that can turn off various features, but I wonder if I can really strip it down further by eliminating parsing, query planning, etc, altogether, and only support the virtual machine.  I do need virtual tables, though.  In my particular use-case, I only need read access -- no create or update.  The thinking being that I can build queries offline and compile them into the p-code (or whatever it's called), and either burn those well know queries into flash, or perhaps send them down the wire as needed.  Then of course (maybe even more critically), can I control ram usage in a deterministic way such that it will still work on memory-constrained devices (e.g. having a total of 128 KiB max for the whole system).

Anway, has this been discussed before?  Or is it a fool's errand?

Cheers!

-dave
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


___________________________________________
 Gunter Hick | Software Engineer | Scientific Games International GmbH | Klitschgasse 2-4, A-1130 Vienna | FN 157284 a, HG Wien, DVR: 0430013 | (O) [phone] - 0

May be privileged. May be confidential. Please delete if not the addressee.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[decalek]

On 2018-04-15 21:54, dave wrote:
>  
> Anway, has this been discussed before?  Or is it a fool's errand?
>

Same kind of fool here :-)

After Mr.Hipp response, which suggest that hwaci.com does not sell SSE
anymore, the community could try to reproduce the same idea against
current VDBE instruction set (ideally with development communication
hosted here, so we can receive advice from the SQLite core team).

Does anybody have an idea for table set (schema) for the
assemble/disassemble functions in question to target?

Kind Regards,
Alek

Beyond concrete project uses and the fun of playing with VDBE code with
SQL :-), I think that disassemble extension will be appropriate for
university curses too.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[dave]

...

> > size -- both code and RAM.  I know about the various
> compile switches that
> > can turn off various features, but I wonder if I can really
> strip it down
> > further by eliminating parsing, query planning, etc,
> altogether, and only
> > support the virtual machine.  I do need virtual tables,
> though.  In my
> > particular use-case, I only need read access -- no create
> or update.  The

...

> > such that it will still work on memory-constrained devices
> (e.g. having a
> > total of 128 KiB max for the whole system).
> >
> > Anway, has this been discussed before?  Or is it a fool's errand?
>
> We did this once, back in 2005, for a startup company in Boston.  It
> was called "SSE".  Unfortunately, we didn't continue to support it.  I
> went looking for the source code and could not find it.
>
> The database was to run on a smart-card with limited RAM.  All of the
> prepared statements were generated on a workstation, then serialized
> and stored in a special table in the database file.  The application
> would then use a special API that would deserialize a prepared
> statement (identified by a well-known integer) then bind parameters
> and run it.
>
> So much has changed in the SQLite bytecode engine since then that
> there is basically zero chance that SSE would still run today, even if
> I could find the source code.
>
> --
> D. Richard Hipp
> [hidden email]

Ah, groovy.  Well, at least that is validation of the concept.  So it sounds
like I have a side project for my copious free time!

-dave


_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[dave]

Thanks for the feedback.  The schema would indeed be fixed.  It sounds like
I have an 'interesting' side project in my future!

> -----Original Message-----
> From: sqlite-users
> [mailto:[hidden email]] On
> Behalf Of Hick Gunter
> Sent: Monday, April 16, 2018 1:50 AM
> To: 'SQLite mailing list'
> Subject: Re: [sqlite] [EXTERNAL] kooky thought: a vm-only
> build (for embedded). feasible?
>
>
> There have been some inquries on the list into executing
> pre-generated bytecode. If you have a fixed schema you may be
> able to create a valid sqlite3_stmt pointer from bytecode
> stored somewhere.
>
> -----Ursprüngliche Nachricht-----
> Von: sqlite-users
> [mailto:[hidden email]] Im
> Auftrag von dave
> Gesendet: Sonntag, 15. April 2018 20:55
> An: [hidden email]
> Betreff: [EXTERNAL] [sqlite] kooky thought: a vm-only build
> (for embedded). feasible?
>
> I had a stray thought, and wanted to ask if it's been thunk
> before,and if so what is the thinking?  Or just for commentary.
>
> I have been building a system, part of which uses sqlite and
> virtual tables.
> This is working great in a desktop/mobile environment.  
> However, eventually one day, I will want to migrate aspects
> of the product to deeply embedded systems (e.g. something
> like an STM32F4 class chip), and am thinking about size --
> both code and RAM.  I know about the various compile switches
> that can turn off various features, but I wonder if I can
> really strip it down further by eliminating parsing, query
> planning, etc, altogether, and only support the virtual
> machine.  I do need virtual tables, though.  In my particular
> use-case, I only need read access -- no create or update.  
> The thinking being that I can build queries offline and
> compile them into the p-code (or whatever it's called), and
> either burn those well know queries into flash, or perhaps
> send them down the wire as needed.  Then of course (maybe
> even more critically), can I control ram usage in a
> deterministic way such that it will still work on
> memory-constrained devices (e.g. having a total of 128 KiB
> max for the whole system).
>
> Anway, has this been discussed before?  Or is it a fool's errand?
>
> Cheers!
>
> -dave
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
>
> ___________________________________________
>  Gunter Hick | Software Engineer | Scientific Games
> International GmbH | Klitschgasse 2-4, A-1130 Vienna | FN
> 157284 a, HG Wien, DVR: 0430013 | (O) [phone] - 0
>
> May be privileged. May be confidential. Please delete if not
> the addressee.
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users