Pascal (almost) style BLOBs

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Pascal (almost) style BLOBs

William Drago-2
All,

Any thoughts on using the first byte of a BLOB to indicate
what kind of data that BLOB contains?

For example:
0 = plain old bytes
1 = 16 bit integers
2 = 32 bit integers
3 = singles
4 = doubles, etc.

I am forced to consider this approach because my function
will be receiving blobs but will have no idea what those
blobs are. However, I can ask the sender to prepend the blob
with an indicator byte to provide the necessary information.

In the past I have used comments in the table structure and
even used custom types (e.g. i16BLOB, for a blob that
contains 16 bit ints), but in this case I will not have
access to that information. The data is coming to me in the
form of an ADO.NET DataTable which does not contain such
information.

Of course tagging the blobs like this will increase the size
of the database, but not by much, and if this "feature" is
not clearly documented someone in the future will have a
very hard time figuring out why the blobs don't make sense.

Is there anything else I should be aware of?

Thanks,
Bill






_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Pascal (almost) style BLOBs

Ketil Froyn
I'm not sure I understand why you can't specify the type in a separate
column, rather than prepending it to the blob? That seems like a more
flexible way to have access to the information.

Regards, Ketil
On 17 Apr 2016 2:02 p.m., "William Drago" <[hidden email]> wrote:

> All,
>
> Any thoughts on using the first byte of a BLOB to indicate what kind of
> data that BLOB contains?
>
> For example:
> 0 = plain old bytes
> 1 = 16 bit integers
> 2 = 32 bit integers
> 3 = singles
> 4 = doubles, etc.
>
> I am forced to consider this approach because my function will be
> receiving blobs but will have no idea what those blobs are. However, I can
> ask the sender to prepend the blob with an indicator byte to provide the
> necessary information.
>
> In the past I have used comments in the table structure and even used
> custom types (e.g. i16BLOB, for a blob that contains 16 bit ints), but in
> this case I will not have access to that information. The data is coming to
> me in the form of an ADO.NET DataTable which does not contain such
> information.
>
> Of course tagging the blobs like this will increase the size of the
> database, but not by much, and if this "feature" is not clearly documented
> someone in the future will have a very hard time figuring out why the blobs
> don't make sense.
>
> Is there anything else I should be aware of?
>
> Thanks,
> Bill
>
>
>
>
>
>
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Pascal (almost) style BLOBs

Simon Slavin-3
In reply to this post by William Drago-2

On 17 Apr 2016, at 1:02pm, William Drago <[hidden email]> wrote:

> I am forced to consider this approach because my function will be receiving blobs but will have no idea what those blobs are.

Why do you care ?  A BLOB is a BLOB.  It doesn't matter whether it's an integer, a string, a car, a peace treaty or the smell of newly cut grass.  You store it when told to and retrieve it when told to.

> if this "feature" is not clearly documented someone in the future will have a very hard time figuring out why the blobs don't make sense.

BLOBs don't make sense.  They're opaque.  You are not meant to understand anything about what's in a BLOB.  That's for the layer of your software which packs the information into a blob or unpacks the blob back into information.

If you actually want to store, for example, integers, you shouldn't be storing a BLOB in the first place.

Simon.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Pascal (almost) style BLOBs

Brad Stiles-2
In reply to this post by William Drago-2
An ADO.NET DataTable *can* have type information for table columns, if you add a parameter to the WriteXml method to have it write the schema along with the data. Obviously, *their* data must be typed from their source, or be inferred, for this to work.

See if your provider can do this for you. If they can, you should have little problem reading it, assuming you're not using .Net yourself.

> On Apr 17, 2016, at 07:02, William Drago <[hidden email]> wrote:
>
> All,
>
> Any thoughts on using the first byte of a BLOB to indicate what kind of data that BLOB contains?
>
> For example:
> 0 = plain old bytes
> 1 = 16 bit integers
> 2 = 32 bit integers
> 3 = singles
> 4 = doubles, etc.
>
> I am forced to consider this approach because my function will be receiving blobs but will have no idea what those blobs are. However, I can ask the sender to prepend the blob with an indicator byte to provide the necessary information.
>
> In the past I have used comments in the table structure and even used custom types (e.g. i16BLOB, for a blob that contains 16 bit ints), but in this case I will not have access to that information. The data is coming to me in the form of an ADO.NET DataTable which does not contain such information.
>
> Of course tagging the blobs like this will increase the size of the database, but not by much, and if this "feature" is not clearly documented someone in the future will have a very hard time figuring out why the blobs don't make sense.
>
> Is there anything else I should be aware of?
>
> Thanks,
> Bill
>
>
>
>
>
>
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Pascal (almost) style BLOBs

William Drago-3
In reply to this post by Ketil Froyn
On 4/17/2016 9:20 AM, Ketil Froyn wrote:
> I'm not sure I understand why you can't specify the type in a separate
> column, rather than prepending it to the blob? That seems like a more
> flexible way to have access to the information.

I have no control over the database schema, table structure,
etc.. I also have no control of the select statement that is
generating that data.

-Bill

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Pascal (almost) style BLOBs

William Drago-3
In reply to this post by Simon Slavin-3
On 4/17/2016 9:45 AM, Simon Slavin wrote:
> On 17 Apr 2016, at 1:02pm, William Drago <[hidden email]> wrote:
>
>> I am forced to consider this approach because my function will be receiving blobs but will have no idea what those blobs are.
> Why do you care ?  A BLOB is a BLOB.  It doesn't matter whether it's an integer, a string, a car, a peace treaty or the smell of newly cut grass.  You store it when told to and retrieve it when told to.

My job is to find all the blob columns and unblob them into
arrays of whatever type they originally were before they
were blob'd.

>> if this "feature" is not clearly documented someone in the future will have a very hard time figuring out why the blobs don't make sense.
> BLOBs don't make sense.  They're opaque.  You are not meant to understand anything about what's in a BLOB.  That's for the layer of your software which packs the information into a blob or unpacks the blob back into information.

Understood. I'm writing the code that does the packing and
unpacking. The users are designing their own databases. When
they store for example, an array of 16 bit integers, or an
array of 64 bit floats, I want to give them the same thing
back when they query the database not just an array of bytes
that they have to decode themselves.

Since I have control over how the blobs are generated, I can
easily add a type identifier to each blob so that I know how
to unblob it.

-Bill
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Pascal (almost) style BLOBs

William Drago-2
In reply to this post by Ketil Froyn


On 4/17/2016 9:20 AM, Ketil Froyn wrote:
> I'm not sure I understand why you can't specify the type in a separate
> column, rather than prepending it to the blob? That seems like a more
> flexible way to have access to the information.
I have no control over the database schema, table structure,
etc.. I also have no control of the select statement that is
generating that data.

-Bill
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Pascal (almost) style BLOBs

William Drago-2
In reply to this post by Simon Slavin-3
On 4/17/2016 9:45 AM, Simon Slavin wrote:
> On 17 Apr 2016, at 1:02pm, William Drago <[hidden email]> wrote:
>
>> I am forced to consider this approach because my function will be receiving blobs but will have no idea what those blobs are.
> Why do you care ?  A BLOB is a BLOB.  It doesn't matter whether it's an integer, a string, a car, a peace treaty or the smell of newly cut grass.  You store it when told to and retrieve it when told to.
My job is to find all the blob columns and unblob them into
arrays of whatever type they originally were before they
were blob'd.
>> if this "feature" is not clearly documented someone in the future will have a very hard time figuring out why the blobs don't make sense.
> BLOBs don't make sense.  They're opaque.  You are not meant to understand anything about what's in a BLOB.  That's for the layer of your software which packs the information into a blob or unpacks the blob back into information.
Understood. I'm writing the code that does the packing and
unpacking. The users are designing their own databases. When
they store for example, an array of 16 bit integers, or an
array of 64 bit floats, I want to give them the same thing
back when they query the database not just an array of bytes
that they have to decode themselves.

Since I have control over how the blobs are generated, I can
easily add a type identifier to each blob so that I know how
to unblob it.

-Bill

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Pascal (almost) style BLOBs

William Drago-2
In reply to this post by Brad Stiles-2
On 4/17/2016 10:13 AM, Brad Stiles wrote:
> An ADO.NET DataTable *can* have type information for table columns, if you add a parameter to the WriteXml method to have it write the schema along with the data. Obviously, *their* data must be typed from their source, or be inferred, for this to work.

If the DataTable is the result of a SELECT that involves
more than one table is the schema information included for
each of the tables?

> See if your provider can do this for you. If they can, you should have little problem reading it, assuming you're not using .Net yourself.

I'm confused by your statement. I am using .NET, that's why
I'm dealing with ADO.NET DataTables.

Thanks,
-Bill
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Pascal (almost) style BLOBs

Paul Sanderson
In reply to this post by William Drago-3
You could use a binary PList or some other form of structured data -
the BPList blob could then contain, ints, floats, strings byte arrays
or even embedded binary plists. As it is a structured format then to
an extent it is self documenting (in terms of the stored data types)
Paul
www.sandersonforensics.com
skype: r3scue193
twitter: @sandersonforens
Tel +44 (0)1326 572786
http://sandersonforensics.com/forum/content.php?195-SQLite-Forensic-Toolkit
-Forensic Toolkit for SQLite
email from a work address for a fully functional demo licence


On 17 April 2016 at 16:27, William Drago <[hidden email]> wrote:

> On 4/17/2016 9:45 AM, Simon Slavin wrote:
>>
>> On 17 Apr 2016, at 1:02pm, William Drago <[hidden email]> wrote:
>>
>>> I am forced to consider this approach because my function will be
>>> receiving blobs but will have no idea what those blobs are.
>>
>> Why do you care ?  A BLOB is a BLOB.  It doesn't matter whether it's an
>> integer, a string, a car, a peace treaty or the smell of newly cut grass.
>> You store it when told to and retrieve it when told to.
>
>
> My job is to find all the blob columns and unblob them into arrays of
> whatever type they originally were before they were blob'd.
>
>>> if this "feature" is not clearly documented someone in the future will
>>> have a very hard time figuring out why the blobs don't make sense.
>>
>> BLOBs don't make sense.  They're opaque.  You are not meant to understand
>> anything about what's in a BLOB.  That's for the layer of your software
>> which packs the information into a blob or unpacks the blob back into
>> information.
>
>
> Understood. I'm writing the code that does the packing and unpacking. The
> users are designing their own databases. When they store for example, an
> array of 16 bit integers, or an array of 64 bit floats, I want to give them
> the same thing back when they query the database not just an array of bytes
> that they have to decode themselves.
>
> Since I have control over how the blobs are generated, I can easily add a
> type identifier to each blob so that I know how to unblob it.
>
> -Bill
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Pascal (almost) style BLOBs

William Drago-2
On 4/18/2016 6:46 AM, Paul Sanderson wrote:
> You could use a binary PList or some other form of structured data -
> the BPList blob could then contain, ints, floats, strings byte arrays
> or even embedded binary plists. As it is a structured format then to
> an extent it is self documenting (in terms of the stored data types)
> Paul

Of course! I don't have to be that complicated though, do I?
I've pretty well convinced myself that adding a type
indicator byte to the blob will provide all the information
I need to unblob it properly. I haven't heard any major
objections from the list here, so I'm going try it out.

Thanks for the suggestion,
-Bill

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users