I had a stray thought, and wanted to ask if it's been thunk before,and if so
what is the thinking? Or just for commentary. I have been building a system, part of which uses sqlite and virtual tables. This is working great in a desktop/mobile environment. However, eventually one day, I will want to migrate aspects of the product to deeply embedded systems (e.g. something like an STM32F4 class chip), and am thinking about size -- both code and RAM. I know about the various compile switches that can turn off various features, but I wonder if I can really strip it down further by eliminating parsing, query planning, etc, altogether, and only support the virtual machine. I do need virtual tables, though. In my particular use-case, I only need read access -- no create or update. The thinking being that I can build queries offline and compile them into the p-code (or whatever it's called), and either burn those well know queries into flash, or perhaps send them down the wire as needed. Then of course (maybe even more critically), can I control ram usage in a deterministic way such that it will still work on memory-constrained devices (e.g. having a total of 128 KiB max for the whole system). Anway, has this been discussed before? Or is it a fool's errand? Cheers! -dave _______________________________________________ sqlite-users mailing list [hidden email] http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users |
On 15 Apr 2018, at 7:54pm, dave <[hidden email]> wrote: > I wonder if I can really strip it down > further by eliminating parsing, query planning, etc, altogether, and only > support the virtual machine. I wonder what you would find if you looked through the data structure of sqlite3_stmt. Presumably the compilation process would convert the SQL text into bytecode and the bytecode would be stored in the statement. Once you have seen the bytecode from your desired SQL, it might be possible to write a C function which accepts a pointer and a length and creates a statement with a copy of that chunk of memory as the bytecode and everything else set up the way it is in a newly-created statement. Or something like that. Simon. _______________________________________________ sqlite-users mailing list [hidden email] http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users |
That was more-or-less my thinking.
Mostly, my inquiry is to solicit any advice or wisdom-of-the-ages, or even advice against it. After having sent that, I suspect that this would more likely wind up being something I'm on my own in doing the hands-on work, but I still welcome any advice on how to approach the surgery. -dave > -----Original Message----- > From: sqlite-users > [mailto:[hidden email]] On > Behalf Of Simon Slavin > Sent: Sunday, April 15, 2018 2:06 PM > To: SQLite mailing list > Subject: Re: [sqlite] kooky thought: a vm-only build (for > embedded).feasible? > > > > > On 15 Apr 2018, at 7:54pm, dave <[hidden email]> wrote: > > > I wonder if I can really strip it down > > further by eliminating parsing, query planning, etc, > altogether, and only > > support the virtual machine. > > I wonder what you would find if you looked through the data > structure of sqlite3_stmt. Presumably the compilation > process would convert the SQL text into bytecode and the > bytecode would be stored in the statement. > > Once you have seen the bytecode from your desired SQL, it > might be possible to write a C function which accepts a > pointer and a length and creates a statement with a copy of > that chunk of memory as the bytecode and everything else set > up the way it is in a newly-created statement. > > Or something like that. > > Simon. > _______________________________________________ > sqlite-users mailing list > [hidden email] > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > _______________________________________________ sqlite-users mailing list [hidden email] http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users |
In reply to this post by dave
On 4/15/18, dave <[hidden email]> wrote:
> I had a stray thought, and wanted to ask if it's been thunk before,and if so > what is the thinking? Or just for commentary. > > I have been building a system, part of which uses sqlite and virtual tables. > This is working great in a desktop/mobile environment. However, eventually > one day, I will want to migrate aspects of the product to deeply embedded > systems (e.g. something like an STM32F4 class chip), and am thinking about > size -- both code and RAM. I know about the various compile switches that > can turn off various features, but I wonder if I can really strip it down > further by eliminating parsing, query planning, etc, altogether, and only > support the virtual machine. I do need virtual tables, though. In my > particular use-case, I only need read access -- no create or update. The > thinking being that I can build queries offline and compile them into the > p-code (or whatever it's called), and either burn those well know queries > into flash, or perhaps send them down the wire as needed. Then of course > (maybe even more critically), can I control ram usage in a deterministic way > such that it will still work on memory-constrained devices (e.g. having a > total of 128 KiB max for the whole system). > > Anway, has this been discussed before? Or is it a fool's errand? We did this once, back in 2005, for a startup company in Boston. It was called "SSE". Unfortunately, we didn't continue to support it. I went looking for the source code and could not find it. The database was to run on a smart-card with limited RAM. All of the prepared statements were generated on a workstation, then serialized and stored in a special table in the database file. The application would then use a special API that would deserialize a prepared statement (identified by a well-known integer) then bind parameters and run it. So much has changed in the SQLite bytecode engine since then that there is basically zero chance that SSE would still run today, even if I could find the source code. -- D. Richard Hipp [hidden email] _______________________________________________ sqlite-users mailing list [hidden email] http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users |
In reply to this post by dave
There have been some inquries on the list into executing pre-generated bytecode. If you have a fixed schema you may be able to create a valid sqlite3_stmt pointer from bytecode stored somewhere.
-----Ursprüngliche Nachricht----- Von: sqlite-users [mailto:[hidden email]] Im Auftrag von dave Gesendet: Sonntag, 15. April 2018 20:55 An: [hidden email] Betreff: [EXTERNAL] [sqlite] kooky thought: a vm-only build (for embedded). feasible? I had a stray thought, and wanted to ask if it's been thunk before,and if so what is the thinking? Or just for commentary. I have been building a system, part of which uses sqlite and virtual tables. This is working great in a desktop/mobile environment. However, eventually one day, I will want to migrate aspects of the product to deeply embedded systems (e.g. something like an STM32F4 class chip), and am thinking about size -- both code and RAM. I know about the various compile switches that can turn off various features, but I wonder if I can really strip it down further by eliminating parsing, query planning, etc, altogether, and only support the virtual machine. I do need virtual tables, though. In my particular use-case, I only need read access -- no create or update. The thinking being that I can build queries offline and compile them into the p-code (or whatever it's called), and either burn those well know queries into flash, or perhaps send them down the wire as needed. Then of course (maybe even more critically), can I control ram usage in a deterministic way such that it will still work on memory-constrained devices (e.g. having a total of 128 KiB max for the whole system). Anway, has this been discussed before? Or is it a fool's errand? Cheers! -dave _______________________________________________ sqlite-users mailing list [hidden email] http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users ___________________________________________ Gunter Hick | Software Engineer | Scientific Games International GmbH | Klitschgasse 2-4, A-1130 Vienna | FN 157284 a, HG Wien, DVR: 0430013 | (O) +43 1 80100 - 0 May be privileged. May be confidential. Please delete if not the addressee. _______________________________________________ sqlite-users mailing list [hidden email] http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users |
In reply to this post by dave
On 2018-04-15 21:54, dave wrote:
> > Anway, has this been discussed before? Or is it a fool's errand? > Same kind of fool here :-) After Mr.Hipp response, which suggest that hwaci.com does not sell SSE anymore, the community could try to reproduce the same idea against current VDBE instruction set (ideally with development communication hosted here, so we can receive advice from the SQLite core team). Does anybody have an idea for table set (schema) for the assemble/disassemble functions in question to target? Kind Regards, Alek Beyond concrete project uses and the fun of playing with VDBE code with SQL :-), I think that disassemble extension will be appropriate for university curses too. _______________________________________________ sqlite-users mailing list [hidden email] http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users |
In reply to this post by Richard Hipp-3
...
> > size -- both code and RAM. I know about the various > compile switches that > > can turn off various features, but I wonder if I can really > strip it down > > further by eliminating parsing, query planning, etc, > altogether, and only > > support the virtual machine. I do need virtual tables, > though. In my > > particular use-case, I only need read access -- no create > or update. The > > such that it will still work on memory-constrained devices > (e.g. having a > > total of 128 KiB max for the whole system). > > > > Anway, has this been discussed before? Or is it a fool's errand? > > We did this once, back in 2005, for a startup company in Boston. It > was called "SSE". Unfortunately, we didn't continue to support it. I > went looking for the source code and could not find it. > > The database was to run on a smart-card with limited RAM. All of the > prepared statements were generated on a workstation, then serialized > and stored in a special table in the database file. The application > would then use a special API that would deserialize a prepared > statement (identified by a well-known integer) then bind parameters > and run it. > > So much has changed in the SQLite bytecode engine since then that > there is basically zero chance that SSE would still run today, even if > I could find the source code. > > -- > D. Richard Hipp > [hidden email] Ah, groovy. Well, at least that is validation of the concept. So it sounds like I have a side project for my copious free time! -dave _______________________________________________ sqlite-users mailing list [hidden email] http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users |
In reply to this post by Hick Gunter
Thanks for the feedback. The schema would indeed be fixed. It sounds like
I have an 'interesting' side project in my future! > -----Original Message----- > From: sqlite-users > [mailto:[hidden email]] On > Behalf Of Hick Gunter > Sent: Monday, April 16, 2018 1:50 AM > To: 'SQLite mailing list' > Subject: Re: [sqlite] [EXTERNAL] kooky thought: a vm-only > build (for embedded). feasible? > > > There have been some inquries on the list into executing > pre-generated bytecode. If you have a fixed schema you may be > able to create a valid sqlite3_stmt pointer from bytecode > stored somewhere. > > -----Ursprüngliche Nachricht----- > Von: sqlite-users > [mailto:[hidden email]] Im > Auftrag von dave > Gesendet: Sonntag, 15. April 2018 20:55 > An: [hidden email] > Betreff: [EXTERNAL] [sqlite] kooky thought: a vm-only build > (for embedded). feasible? > > I had a stray thought, and wanted to ask if it's been thunk > before,and if so what is the thinking? Or just for commentary. > > I have been building a system, part of which uses sqlite and > virtual tables. > This is working great in a desktop/mobile environment. > However, eventually one day, I will want to migrate aspects > of the product to deeply embedded systems (e.g. something > like an STM32F4 class chip), and am thinking about size -- > both code and RAM. I know about the various compile switches > that can turn off various features, but I wonder if I can > really strip it down further by eliminating parsing, query > planning, etc, altogether, and only support the virtual > machine. I do need virtual tables, though. In my particular > use-case, I only need read access -- no create or update. > The thinking being that I can build queries offline and > compile them into the p-code (or whatever it's called), and > either burn those well know queries into flash, or perhaps > send them down the wire as needed. Then of course (maybe > even more critically), can I control ram usage in a > deterministic way such that it will still work on > memory-constrained devices (e.g. having a total of 128 KiB > max for the whole system). > > Anway, has this been discussed before? Or is it a fool's errand? > > Cheers! > > -dave > _______________________________________________ > sqlite-users mailing list > [hidden email] > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > > > ___________________________________________ > Gunter Hick | Software Engineer | Scientific Games > International GmbH | Klitschgasse 2-4, A-1130 Vienna | FN > 157284 a, HG Wien, DVR: 0430013 | (O) +43 1 80100 - 0 > > May be privileged. May be confidential. Please delete if not > the addressee. > _______________________________________________ > sqlite-users mailing list > [hidden email] > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > _______________________________________________ sqlite-users mailing list [hidden email] http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users |
Free forum by Nabble | Edit this page |