readfile/writefile extensioln, and UTF8 on Windows....

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

readfile/writefile extensioln, and UTF8 on Windows....

dave
Folks;  I recently had some trouble using the readfile() extension until I
noticed it was due to there being filenames with Unicode in them, and that
the existing implementation using fopen() doesn't do UTF8 on Windows (I
think it uses the ambient code page).
 
I modified the extension code to support UTF8 filenames on Windows, and it
works fine (at least in my test cases!).  Thinking that someone might fine
it useful, I am attaching the modified code.  Use it if and however you
like.  Also note there is an embedded copy of fileio.c in shell.c that also
needs the mod, for the sqlite shell.c builds.
 
Attached herewith.
 
Cheers!
 
-dave
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: readfile/writefile extensioln, and UTF8 on Windows....

dave
Welp; I guess attachements get stripped off.  And maybe I should have sent
this to the 'dev' list anyway, so I'll try there.

> -----Original Message-----
...
> To: 'SQLite mailing list'
> Subject: [sqlite] readfile/writefile extensioln, and UTF8 on
> Windows....
>
> Folks;  I recently had some trouble using the readfile()
...
>  
> Attached herewith.
>  
> Cheers!
>  
> -dave


_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: readfile/writefile extensioln, and UTF8 on Windows....

Dan Kennedy-4
In reply to this post by dave
On 12/12/2017 12:42 AM, dave wrote:

> Folks;  I recently had some trouble using the readfile() extension until I
> noticed it was due to there being filenames with Unicode in them, and that
> the existing implementation using fopen() doesn't do UTF8 on Windows (I
> think it uses the ambient code page).
>  
> I modified the extension code to support UTF8 filenames on Windows, and it
> works fine (at least in my test cases!).  Thinking that someone might fine
> it useful, I am attaching the modified code.  Use it if and however you
> like.  Also note there is an embedded copy of fileio.c in shell.c that also
> needs the mod, for the sqlite shell.c builds.


shell.c is a generated file, so the build process will pull in changes
to ext/misc/fileio.c.

The mailing list has discarded your attachment though. Can you post the
patch inline or upload it somewhere and post a link?

Thanks,
Dan.



>  
> Attached herewith.
>  
> Cheers!
>  
> -dave
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: readfile/writefile extensioln, and UTF8 on Windows....

dave
Certainly; the source file is short.  I did successfully post it to the
'dev' list, however, which I guess is set to accept attachements.

You might want to make edits if you do choose to encorporate it into the
canonical codebase.  E.g.:
*  I think there are some existing sqlite utility functions for doing utf8
to utf16 that you might prefer instead of the MultiByteToWideChar Windows
native function
*  maybe you prefer wfopen to CreateFileW (I think that would work also)
*  maybe there is a better #define for windows conditional compilation in
sqlite codebase that you might prefer to the ones I used
*  etc.

Cheers,

-dave

//fileio.patch start  ===================================


18a19,21
> #if defined(WIN32) || defined(WIN64) || defined(WINDOWS)
> #include <windows.h>
> #endif
29a33,73

> #if defined(WIN32) || defined(WIN64) || defined(WINDOWS)
>   const char* zName;
>   int nCvt;
>   wchar_t* awchName;
>   HANDLE hfile;
>   BY_HANDLE_FILE_INFORMATION finfo;
>   void* pBuf;
>   DWORD dwRead;
>
>   (void)(argc);  /* Unused parameter */
>   /*get file name (utf8)*/
>   zName = (const char*)sqlite3_value_text( argv[0] );
>   if ( zName == 0 ) return;
>   /*figure out how many (utf-16)*/
>   nCvt = MultiByteToWideChar( CP_UTF8, 0, zName, -1, NULL, 0 );
>   if ( 0 == nCvt ) return;
>   awchName = sqlite3_malloc( nCvt * sizeof( wchar_t ) );
>   nCvt = MultiByteToWideChar( CP_UTF8, 0, zName, -1, awchName, nCvt );
>   hfile = CreateFileW( awchName, GENERIC_READ, FILE_SHARE_READ|
FILE_SHARE_DELETE, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL );

>   sqlite3_free( awchName );
>   if ( INVALID_HANDLE_VALUE == hfile ) return;
>   if ( !GetFileInformationByHandle( hfile, &finfo ) )
>   {
>     CloseHandle( hfile );
>     return;
>   }
>   pBuf = sqlite3_malloc( (int)finfo.nFileSizeLow );
>   if ( NULL == pBuf )
>   {
>     CloseHandle( hfile );
>     return;
>   }
>   if ( !ReadFile( hfile, pBuf, finfo.nFileSizeLow, &dwRead, NULL ) )
>   {
>     sqlite3_free( pBuf );
>     CloseHandle( hfile );
>     return;
>   }
>   sqlite3_result_blob( context, pBuf, dwRead, sqlite3_free );
>   CloseHandle( hfile );
> #else
49a94
> #endif
62a108,149

> #if defined(WIN32) || defined(WIN64) || defined(WINDOWS)
>   const char* zName;
>   int nCvt;
>   wchar_t* awchName;
>   HANDLE hfile;
>   int nLen;
>   const char* z;
>   sqlite3_int64 rc;
>   DWORD dwWritten;
>
>   (void)(argc);  /* Unused parameter */
>   //get file name (utf8)
>   zName = (const char*)sqlite3_value_text( argv[0] );
>   if ( zName == 0 ) return;
>   //figure out how many (utf-16)
>   nCvt = MultiByteToWideChar( CP_UTF8, 0, zName, -1, NULL, 0 );
>   if ( 0 == nCvt ) return;
>   awchName = sqlite3_malloc( nCvt * sizeof( wchar_t ) );
>   nCvt = MultiByteToWideChar( CP_UTF8, 0, zName, -1, awchName, nCvt );
>   hfile = CreateFileW( awchName, GENERIC_WRITE, FILE_SHARE_WRITE |
FILE_SHARE_DELETE, NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL );

>   sqlite3_free( awchName );
>   if ( INVALID_HANDLE_VALUE == hfile ) return;
>   nLen = sqlite3_value_bytes( argv[1] );
>   z = (const char*)sqlite3_value_blob( argv[1] );
>   if ( NULL == z )
>   {
>     rc = 0;
>   }
>   else
>   {
>     if ( !WriteFile( hfile, z, (DWORD)nLen, &dwWritten, NULL ) )
>     {
>       rc = 0;
>     }
>     else
>     {
>       rc = (sqlite3_int64)dwWritten;
>     }
>   }
>   CloseHandle( hfile );
>   sqlite3_result_int64( context, rc );
> #else
80a168
> #endif


//fileio.patch end===============================================

> -----Original Message-----
> From: sqlite-users
> [mailto:[hidden email]] On
> Behalf Of Dan Kennedy
> Sent: Monday, December 11, 2017 2:09 PM
> To: [hidden email]
> Subject: Re: [sqlite] readfile/writefile extensioln, and UTF8
> on Windows....
>
>
> On 12/12/2017 12:42 AM, dave wrote:
> > Folks;  I recently had some trouble using the readfile()
> extension until I
> > noticed it was due to there being filenames with Unicode in
> them, and that
> > the existing implementation using fopen() doesn't do UTF8
> on Windows (I
> > think it uses the ambient code page).
> >  
> > I modified the extension code to support UTF8 filenames on
> Windows, and it
> > works fine (at least in my test cases!).  Thinking that
> someone might fine
> > it useful, I am attaching the modified code.  Use it if and
> however you
> > like.  Also note there is an embedded copy of fileio.c in
> shell.c that also
> > needs the mod, for the sqlite shell.c builds.
>
>
> shell.c is a generated file, so the build process will pull
> in changes
> to ext/misc/fileio.c.
>
> The mailing list has discarded your attachment though. Can
> you post the
> patch inline or upload it somewhere and post a link?
>
> Thanks,
> Dan.
>
>
>
> >  
> > Attached herewith.
> >  
> > Cheers!
> >  
> > -dave
> > _______________________________________________
> > sqlite-users mailing list
> > [hidden email]
> > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
>
> _______________________________________________
> sqlite-users mailing list
> [hidden email]
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>


_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users