Unicode quote characters parsing in FTS5 queries

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Unicode quote characters parsing in FTS5 queries

David Guillen Fandos
Hello there,

I was running some queries using FTS5 in my database and I generally
sanitize the input by removing non-ascii characters, except for all the
unicode chars above 128 or so. That way I get rid of stuff like quotes,
double quotes, periods, etc.

However my queries where choking on some input that contained the U+2018
and U+2019 characters. It seems like these chars are being treated like
a regular quote character (I get the same error as if I replaced it with
a regular quote character). Is this intended behaviour? The error I'm
getting is "Result: fts5: phrase queries are not supported
(detail!=full" (and a SQLITE_ERROR calling step()).

I'm quite confused with the behaviour I'm seeing here. Any guidance is
appreciated.

Thanks

David
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: Unicode quote characters parsing in FTS5 queries

Simon Slavin-3
On 3 Feb 2020, at 9:30pm, David Guillen Fandos <[hidden email]> wrote:

> However my queries where choking on some input that contained the U+2018 and U+2019 characters. It seems like these chars are being treated like a regular quote character

At least one version of SQL respects those characters for quoting strings.  Can't remember which one at the moment.  I suspect that the problem you're encountering is intentional.

One test you can try is to double that character (I'm typing the code, you type the character)

'abcU+2018U+2018def'

and see whether this is still a syntax error, and whether you get zero, one or two of those characters in the output string.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users