No unicode characters in sqlite3 console prg

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

No unicode characters in sqlite3 console prg

larshgf
Hello,

 

Is this an error of my windows 10 cmd prompt or is it a general problem that
the sqlite3 console is not able to show unicode in the cmd-window.??

No problem writing to the database and show (greek) unicode characters in
the cmd prompt as I key them, but when I want to show the data with the SQL
SELECT * FROM Gloser then the greek word is shown with most of its letters
as questionmarks.

It is not necessarily a problem as in my dbgrid of my program the unicode is
shown correctly. I am just curious!

 

Best Regards Lars

 

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: No unicode characters in sqlite3 console prg

Tim Streater-3
On 11 Aug 2018, at 17:51, Lars Frederiksen <[hidden email]> wrote:

> Is this an error of my windows 10 cmd prompt or is it a general problem that
> the sqlite3 console is not able to show unicode in the cmd-window.??
>
> No problem writing to the database and show (greek) unicode characters in
> the cmd prompt as I key them, but when I want to show the data with the SQL
> SELECT * FROM Gloser then the greek word is shown with most of its letters
> as questionmarks.

Using macOS Mavericks (10.9.5):

SQLite version 3.14.0 2016-08-08 13:40:27
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> create table x (q text);
sqlite> insert into x values ('∑ß∂π');
sqlite> select * from x;
q     
----------
∑ß∂π


Seems to work.



--
Cheers  --  Tim
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: No unicode characters in sqlite3 console prg

Simon Slavin-3
In reply to this post by larshgf
On 11 Aug 2018, at 5:51pm, Lars Frederiksen <[hidden email]> wrote:

> Is this an error of my windows 10 cmd prompt or is it a general problem that
> the sqlite3 console is not able to show unicode in the cmd-window.??

It's a Windows thing.  But I don't know how to fix it in Windows 10.  Does Windows 10 still use code pages in the Console ?

Try executing this command at the Console prompt before doing stuff with sqlite3:

chcp 65001

Does this make things work properly ?  If not, I would google the following string:

"windows 10" "code page" console

Simon.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: No unicode characters in sqlite3 console prg

Rowan Worth-2
In reply to this post by larshgf
On 12 August 2018 at 00:51, Lars Frederiksen <[hidden email]> wrote:

> Is this an error of my windows 10 cmd prompt or is it a general problem
> that
> the sqlite3 console is not able to show unicode in the cmd-window.??
>
> No problem writing to the database and show (greek) unicode characters in
> the cmd prompt as I key them, but when I want to show the data with the SQL
> SELECT * FROM Gloser then the greek word is shown with most of its letters
> as questionmarks.
>

What version of sqlite are you using? There was significant efforts to
improve windows console unicode handling awhile back, which I think made it
in for 3.12/3.13.

If you're still seeing issues with a recent sqlite can you provide a short
example which reproduces the issue?
-Rowan
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: [EXTERNAL] No unicode characters in sqlite3 console prg

Hick Gunter
In reply to this post by larshgf
SQLite will return exactly what you give it. Typically, "double converting" from ISO to UTF will mess up your encoding, as will attempting to display UTF on a console that speaks ISO.

-----Ursprüngliche Nachricht-----
Von: sqlite-users [mailto:[hidden email]] Im Auftrag von Lars Frederiksen
Gesendet: Samstag, 11. August 2018 18:52
An: [hidden email]
Betreff: [EXTERNAL] [sqlite] No unicode characters in sqlite3 console prg

Hello,



Is this an error of my windows 10 cmd prompt or is it a general problem that the sqlite3 console is not able to show unicode in the cmd-window.??

No problem writing to the database and show (greek) unicode characters in the cmd prompt as I key them, but when I want to show the data with the SQL SELECT * FROM Gloser then the greek word is shown with most of its letters as questionmarks.

It is not necessarily a problem as in my dbgrid of my program the unicode is shown correctly. I am just curious!



Best Regards Lars



_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


___________________________________________
 Gunter Hick | Software Engineer | Scientific Games International GmbH | Klitschgasse 2-4, A-1130 Vienna | FN 157284 a, HG Wien, DVR: 0430013 | (O) +43 1 80100 - 0

May be privileged. May be confidential. Please delete if not the addressee.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: No unicode characters in sqlite3 console prg

larshgf
In reply to this post by Rowan Worth-2
I use versio 3.19

Here is a print from cmd:

SQLite version 3.19.3 2017-06-08 14:26:16
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> .open flashcard.db
sqlite> select * from Gloser;
2|taß???a ?|taverna|[01_16]
3|taß???a ?|taverna|[01_16]
4|taß???a ?|taverna|[01_16]
5|taß???a ?|taverna|[01_16]
16|taß???a ?|taverna|[01_16]
17|taß???a ?|taverna|[01_16]
18|taß???a ?|taverna|[01_16]
19|de µ?? ?e?...|sig mig engang. (egl "du fortæller mig ikke")|[01_08]
20|d?µ?s?????f?? ?/?|journalist|[01_08]
21|???µa t?|mønt, småmønt|[01_08]
22|??p??a ?|pige|[01_08]
23|µa??t??a ?|elev, skoleelev (kvindelig)|[01_08]
24|µa??t?? ?|elev, skoleelev (mandlig)|[01_08]
25|µp?????? t?|flaske|[01_08]
26|?p??|(konj) som, ligesom, sådan som; eftersom|[01_08]
27|p?t??? t?|glas (Både glas og flaske er intetkøn som vin. Øl er hunkøn)|[01_08]
28|p.?.= pa?ade??µat?? ????|f.eks. (for eksempel)|[01_08]
29|taß???a ?|taverna|[01_16]
30|taß???a ?|taverna|[01_16]
31|de µ?? ?e?...|sig mig engang. (egl "du fortæller mig ikke")|[01_08]
32|d?µ?s?????f?? ?/?|journalist|[01_08]
33|???µa t?|mønt, småmønt|[01_08]
34|??p??a ?|pige|[01_08]
35|µa??t??a ?|elev, skoleelev (kvindelig)|[01_08]
36|µa??t?? ?|elev, skoleelev (mandlig)|[01_08]
37|µp?????? t?|flaske|[01_08]
38|?p??|(konj) som, ligesom, sådan som; eftersom|[01_08]
39|p?t??? t?|glas (Både glas og flaske er intetkøn som vin. Øl er hunkøn)|[01_08]
40|p.?.= pa?ade??µat?? ????|f.eks. (for eksempel)|[01_08]
41|a???|men|[01_10]
42|a???|men|[01_10]
43|a???|men|[01_10]
44|a???|men|[01_10]
45|a???|men|[01_10]
46|a???|men|[01_10]
47|a???|men|[01_10]
48|a???|men|[01_10]

The second field should show greek words but only the letters that are identical in danish and greek is shown. The pure greek letters like β, γ, δ, ε, η etc are shown as questionmarks.
Do you want me to send you the database?

Best Regards
Lars

-----Oprindelig meddelelse-----
Fra: sqlite-users [mailto:[hidden email]] På vegne af Rowan Worth
Sendt: 13. august 2018 03:21
Til: SQLite mailing list
Emne: Re: [sqlite] No unicode characters in sqlite3 console prg

On 12 August 2018 at 00:51, Lars Frederiksen <[hidden email]> wrote:

> Is this an error of my windows 10 cmd prompt or is it a general problem
> that
> the sqlite3 console is not able to show unicode in the cmd-window.??
>
> No problem writing to the database and show (greek) unicode characters in
> the cmd prompt as I key them, but when I want to show the data with the SQL
> SELECT * FROM Gloser then the greek word is shown with most of its letters
> as questionmarks.
>

What version of sqlite are you using? There was significant efforts to
improve windows console unicode handling awhile back, which I think made it
in for 3.12/3.13.

If you're still seeing issues with a recent sqlite can you provide a short
example which reproduces the issue?
-Rowan
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: No unicode characters in sqlite3 console prg

Simon Slavin-3
On 13 Aug 2018, at 1:03pm, Lars Frederiksen <[hidden email]> wrote:

> I use versio 3.19

What operating system ?

If you are using Windows, please issue this command before using sqlite3.exe:

chcp 65001

Simon.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: [EXTERNAL] Re: No unicode characters in sqlite3 console prg

Hick Gunter
In reply to this post by larshgf
You need to show hex(<textfield>) before anyone can verify what encoding is stored.

-----Ursprüngliche Nachricht-----
Von: sqlite-users [mailto:[hidden email]] Im Auftrag von Lars Frederiksen
Gesendet: Montag, 13. August 2018 14:03
An: 'SQLite mailing list' <[hidden email]>
Betreff: [EXTERNAL] Re: [sqlite] No unicode characters in sqlite3 console prg

I use versio 3.19

Here is a print from cmd:

SQLite version 3.19.3 2017-06-08 14:26:16 Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> .open flashcard.db
sqlite> select * from Gloser;
2|taß???a ?|taverna|[01_16]
3|taß???a ?|taverna|[01_16]
4|taß???a ?|taverna|[01_16]
5|taß???a ?|taverna|[01_16]
16|taß???a ?|taverna|[01_16]
17|taß???a ?|taverna|[01_16]
18|taß???a ?|taverna|[01_16]
19|de µ?? ?e?...|sig mig engang. (egl "du fortæller mig ikke")|[01_08]
20|d?µ?s?????f?? ?/?|journalist|[01_08]
21|???µa t?|mønt, småmønt|[01_08]
22|??p??a ?|pige|[01_08]
23|µa??t??a ?|elev, skoleelev (kvindelig)|[01_08]
24|µa??t?? ?|elev, skoleelev (mandlig)|[01_08]
25|µp?????? t?|flaske|[01_08]
26|?p??|(konj) som, ligesom, sådan som; eftersom|[01_08]
27|p?t??? t?|glas (Både glas og flaske er intetkøn som vin. Øl er
27|hunkøn)|[01_08]
28|p.?.= pa?ade??µat?? ????|f.eks. (for eksempel)|[01_08]
29|taß???a ?|taverna|[01_16]
30|taß???a ?|taverna|[01_16]
31|de µ?? ?e?...|sig mig engang. (egl "du fortæller mig ikke")|[01_08]
32|d?µ?s?????f?? ?/?|journalist|[01_08]
33|???µa t?|mønt, småmønt|[01_08]
34|??p??a ?|pige|[01_08]
35|µa??t??a ?|elev, skoleelev (kvindelig)|[01_08]
36|µa??t?? ?|elev, skoleelev (mandlig)|[01_08]
37|µp?????? t?|flaske|[01_08]
38|?p??|(konj) som, ligesom, sådan som; eftersom|[01_08]
39|p?t??? t?|glas (Både glas og flaske er intetkøn som vin. Øl er
39|hunkøn)|[01_08]
40|p.?.= pa?ade??µat?? ????|f.eks. (for eksempel)|[01_08]
41|a???|men|[01_10]
42|a???|men|[01_10]
43|a???|men|[01_10]
44|a???|men|[01_10]
45|a???|men|[01_10]
46|a???|men|[01_10]
47|a???|men|[01_10]
48|a???|men|[01_10]

The second field should show greek words but only the letters that are identical in danish and greek is shown. The pure greek letters like β, γ, δ, ε, η etc are shown as questionmarks.
Do you want me to send you the database?

Best Regards
Lars

-----Oprindelig meddelelse-----
Fra: sqlite-users [mailto:[hidden email]] På vegne af Rowan Worth
Sendt: 13. august 2018 03:21
Til: SQLite mailing list
Emne: Re: [sqlite] No unicode characters in sqlite3 console prg

On 12 August 2018 at 00:51, Lars Frederiksen <[hidden email]> wrote:

> Is this an error of my windows 10 cmd prompt or is it a general
> problem that the sqlite3 console is not able to show unicode in the
> cmd-window.??
>
> No problem writing to the database and show (greek) unicode characters
> in the cmd prompt as I key them, but when I want to show the data with
> the SQL SELECT * FROM Gloser then the greek word is shown with most of
> its letters as questionmarks.
>

What version of sqlite are you using? There was significant efforts to improve windows console unicode handling awhile back, which I think made it in for 3.12/3.13.

If you're still seeing issues with a recent sqlite can you provide a short example which reproduces the issue?
-Rowan
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


___________________________________________
 Gunter Hick | Software Engineer | Scientific Games International GmbH | Klitschgasse 2-4, A-1130 Vienna | FN 157284 a, HG Wien, DVR: 0430013 | (O) +43 1 80100 - 0

May be privileged. May be confidential. Please delete if not the addressee.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: No unicode characters in sqlite3 console prg

larshgf
In reply to this post by Simon Slavin-3
Hi Simon,

I use Windows 10 and have tried
chcp 65001
without any luck. Still the same questionmarks
/Lars


-----Oprindelig meddelelse-----
Fra: sqlite-users [mailto:[hidden email]] På vegne af Simon Slavin
Sendt: 13. august 2018 14:06
Til: SQLite mailing list
Emne: Re: [sqlite] No unicode characters in sqlite3 console prg

On 13 Aug 2018, at 1:03pm, Lars Frederiksen <[hidden email]> wrote:

> I use versio 3.19

What operating system ?

If you are using Windows, please issue this command before using sqlite3.exe:

chcp 65001

Simon.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: [EXTERNAL] Re: No unicode characters in sqlite3 console prg

larshgf
In reply to this post by Hick Gunter
Hi Hick Gunter,
I do not know exactly what you mean by this (hex(<textfield>))?
/Lars

-----Oprindelig meddelelse-----
Fra: sqlite-users [mailto:[hidden email]] På vegne af Hick Gunter
Sendt: 13. august 2018 14:07
Til: 'SQLite mailing list'
Emne: Re: [sqlite] [EXTERNAL] Re: No unicode characters in sqlite3 console prg

You need to show hex(<textfield>) before anyone can verify what encoding is stored.

-----Ursprüngliche Nachricht-----
Von: sqlite-users [mailto:[hidden email]] Im Auftrag von Lars Frederiksen
Gesendet: Montag, 13. August 2018 14:03
An: 'SQLite mailing list' <[hidden email]>
Betreff: [EXTERNAL] Re: [sqlite] No unicode characters in sqlite3 console prg

I use versio 3.19

Here is a print from cmd:

SQLite version 3.19.3 2017-06-08 14:26:16 Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> .open flashcard.db
sqlite> select * from Gloser;
2|taß???a ?|taverna|[01_16]
3|taß???a ?|taverna|[01_16]
4|taß???a ?|taverna|[01_16]
5|taß???a ?|taverna|[01_16]
16|taß???a ?|taverna|[01_16]
17|taß???a ?|taverna|[01_16]
18|taß???a ?|taverna|[01_16]
19|de µ?? ?e?...|sig mig engang. (egl "du fortæller mig ikke")|[01_08]
20|d?µ?s?????f?? ?/?|journalist|[01_08]
21|???µa t?|mønt, småmønt|[01_08]
22|??p??a ?|pige|[01_08]
23|µa??t??a ?|elev, skoleelev (kvindelig)|[01_08]
24|µa??t?? ?|elev, skoleelev (mandlig)|[01_08]
25|µp?????? t?|flaske|[01_08]
26|?p??|(konj) som, ligesom, sådan som; eftersom|[01_08]
27|p?t??? t?|glas (Både glas og flaske er intetkøn som vin. Øl er
27|hunkøn)|[01_08]
28|p.?.= pa?ade??µat?? ????|f.eks. (for eksempel)|[01_08]
29|taß???a ?|taverna|[01_16]
30|taß???a ?|taverna|[01_16]
31|de µ?? ?e?...|sig mig engang. (egl "du fortæller mig ikke")|[01_08]
32|d?µ?s?????f?? ?/?|journalist|[01_08]
33|???µa t?|mønt, småmønt|[01_08]
34|??p??a ?|pige|[01_08]
35|µa??t??a ?|elev, skoleelev (kvindelig)|[01_08]
36|µa??t?? ?|elev, skoleelev (mandlig)|[01_08]
37|µp?????? t?|flaske|[01_08]
38|?p??|(konj) som, ligesom, sådan som; eftersom|[01_08]
39|p?t??? t?|glas (Både glas og flaske er intetkøn som vin. Øl er
39|hunkøn)|[01_08]
40|p.?.= pa?ade??µat?? ????|f.eks. (for eksempel)|[01_08]
41|a???|men|[01_10]
42|a???|men|[01_10]
43|a???|men|[01_10]
44|a???|men|[01_10]
45|a???|men|[01_10]
46|a???|men|[01_10]
47|a???|men|[01_10]
48|a???|men|[01_10]

The second field should show greek words but only the letters that are identical in danish and greek is shown. The pure greek letters like β, γ, δ, ε, η etc are shown as questionmarks.
Do you want me to send you the database?

Best Regards
Lars

-----Oprindelig meddelelse-----
Fra: sqlite-users [mailto:[hidden email]] På vegne af Rowan Worth
Sendt: 13. august 2018 03:21
Til: SQLite mailing list
Emne: Re: [sqlite] No unicode characters in sqlite3 console prg

On 12 August 2018 at 00:51, Lars Frederiksen <[hidden email]> wrote:

> Is this an error of my windows 10 cmd prompt or is it a general
> problem that the sqlite3 console is not able to show unicode in the
> cmd-window.??
>
> No problem writing to the database and show (greek) unicode characters
> in the cmd prompt as I key them, but when I want to show the data with
> the SQL SELECT * FROM Gloser then the greek word is shown with most of
> its letters as questionmarks.
>

What version of sqlite are you using? There was significant efforts to improve windows console unicode handling awhile back, which I think made it in for 3.12/3.13.

If you're still seeing issues with a recent sqlite can you provide a short example which reproduces the issue?
-Rowan
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


___________________________________________
 Gunter Hick | Software Engineer | Scientific Games International GmbH | Klitschgasse 2-4, A-1130 Vienna | FN 157284 a, HG Wien, DVR: 0430013 | (O) +43 1 80100 - 0

May be privileged. May be confidential. Please delete if not the addressee.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: [EXTERNAL] Re: No unicode characters in sqlite3 console prg

Hick Gunter
Select graesk, hex(graesk), dansk from gloser;

Check the hex output to see if it conforms to UTF specs; if so, then your console is expecting a different encoding (probably ISO). If not, then your data was probably entered from a source the encodes in ISO.

-----Ursprüngliche Nachricht-----
Von: sqlite-users [mailto:[hidden email]] Im Auftrag von Lars Frederiksen
Gesendet: Montag, 13. August 2018 14:24
An: 'SQLite mailing list' <[hidden email]>
Betreff: Re: [sqlite] [EXTERNAL] Re: No unicode characters in sqlite3 console prg

Hi Hick Gunter,
I do not know exactly what you mean by this (hex(<textfield>))?
/Lars

-----Oprindelig meddelelse-----
Fra: sqlite-users [mailto:[hidden email]] På vegne af Hick Gunter
Sendt: 13. august 2018 14:07
Til: 'SQLite mailing list'
Emne: Re: [sqlite] [EXTERNAL] Re: No unicode characters in sqlite3 console prg

You need to show hex(<textfield>) before anyone can verify what encoding is stored.

-----Ursprüngliche Nachricht-----
Von: sqlite-users [mailto:[hidden email]] Im Auftrag von Lars Frederiksen
Gesendet: Montag, 13. August 2018 14:03
An: 'SQLite mailing list' <[hidden email]>
Betreff: [EXTERNAL] Re: [sqlite] No unicode characters in sqlite3 console prg

I use versio 3.19

Here is a print from cmd:

SQLite version 3.19.3 2017-06-08 14:26:16 Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> .open flashcard.db
sqlite> select * from Gloser;
2|taß???a ?|taverna|[01_16]
3|taß???a ?|taverna|[01_16]
4|taß???a ?|taverna|[01_16]
5|taß???a ?|taverna|[01_16]
16|taß???a ?|taverna|[01_16]
17|taß???a ?|taverna|[01_16]
18|taß???a ?|taverna|[01_16]
19|de µ?? ?e?...|sig mig engang. (egl "du fortæller mig ikke")|[01_08]
20|d?µ?s?????f?? ?/?|journalist|[01_08]
21|???µa t?|mønt, småmønt|[01_08]
22|??p??a ?|pige|[01_08]
23|µa??t??a ?|elev, skoleelev (kvindelig)|[01_08]
24|µa??t?? ?|elev, skoleelev (mandlig)|[01_08]
25|µp?????? t?|flaske|[01_08]
26|?p??|(konj) som, ligesom, sådan som; eftersom|[01_08]
27|p?t??? t?|glas (Både glas og flaske er intetkøn som vin. Øl er
27|hunkøn)|[01_08]
28|p.?.= pa?ade??µat?? ????|f.eks. (for eksempel)|[01_08]
29|taß???a ?|taverna|[01_16]
30|taß???a ?|taverna|[01_16]
31|de µ?? ?e?...|sig mig engang. (egl "du fortæller mig ikke")|[01_08]
32|d?µ?s?????f?? ?/?|journalist|[01_08]
33|???µa t?|mønt, småmønt|[01_08]
34|??p??a ?|pige|[01_08]
35|µa??t??a ?|elev, skoleelev (kvindelig)|[01_08]
36|µa??t?? ?|elev, skoleelev (mandlig)|[01_08]
37|µp?????? t?|flaske|[01_08]
38|?p??|(konj) som, ligesom, sådan som; eftersom|[01_08]
39|p?t??? t?|glas (Både glas og flaske er intetkøn som vin. Øl er
39|hunkøn)|[01_08]
40|p.?.= pa?ade??µat?? ????|f.eks. (for eksempel)|[01_08]
41|a???|men|[01_10]
42|a???|men|[01_10]
43|a???|men|[01_10]
44|a???|men|[01_10]
45|a???|men|[01_10]
46|a???|men|[01_10]
47|a???|men|[01_10]
48|a???|men|[01_10]

The second field should show greek words but only the letters that are identical in danish and greek is shown. The pure greek letters like β, γ, δ, ε, η etc are shown as questionmarks.
Do you want me to send you the database?

Best Regards
Lars

-----Oprindelig meddelelse-----
Fra: sqlite-users [mailto:[hidden email]] På vegne af Rowan Worth
Sendt: 13. august 2018 03:21
Til: SQLite mailing list
Emne: Re: [sqlite] No unicode characters in sqlite3 console prg

On 12 August 2018 at 00:51, Lars Frederiksen <[hidden email]> wrote:

> Is this an error of my windows 10 cmd prompt or is it a general
> problem that the sqlite3 console is not able to show unicode in the
> cmd-window.??
>
> No problem writing to the database and show (greek) unicode characters
> in the cmd prompt as I key them, but when I want to show the data with
> the SQL SELECT * FROM Gloser then the greek word is shown with most of
> its letters as questionmarks.
>

What version of sqlite are you using? There was significant efforts to improve windows console unicode handling awhile back, which I think made it in for 3.12/3.13.

If you're still seeing issues with a recent sqlite can you provide a short example which reproduces the issue?
-Rowan
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


___________________________________________
 Gunter Hick | Software Engineer | Scientific Games International GmbH | Klitschgasse 2-4, A-1130 Vienna | FN 157284 a, HG Wien, DVR: 0430013 | (O) +43 1 80100 - 0

May be privileged. May be confidential. Please delete if not the addressee.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


___________________________________________
 Gunter Hick | Software Engineer | Scientific Games International GmbH | Klitschgasse 2-4, A-1130 Vienna | FN 157284 a, HG Wien, DVR: 0430013 | (O) +43 1 80100 - 0

May be privileged. May be confidential. Please delete if not the addressee.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: [EXTERNAL] Re: No unicode characters in sqlite3 console prg

Simon Slavin-3
In reply to this post by larshgf
On 13 Aug 2018, at 1:23pm, Lars Frederiksen <[hidden email]> wrote:

> I use versio 3.19

> SQLite version 3.19.3 2017-06-08 14:26:16

Lars,

Please download the current version of the shell tool from the "Precompiled Binaries for Windows" -- "sqlite-tools-win32-x86-3240000.zip". section of

<https://sqlite.org/download.html>

Please run the version from this zip instead of 3.19.  You do not have to worry about updating the DLL libraries on your computer, since the shell tool does not use the DLLs.

If this does not fix your problem ...

> I do not know exactly what you mean by this (hex(<textfield>))?

You execute the following commands:

> sqlite> .open flashcard.db
> sqlite> select * from Gloser;
> 2|taß???a ?|taverna|[01_16]
> 3|taß???a ?|taverna|[01_16]
> 4|taß???a ?|taverna|[01_16]
> 5|taß???a ?|taverna|[01_16]

We want to know what is really in those fields.  If a column in your Gloser table is called "sample" the command we want would be

    SELECT sample,hex(sample) FROM Gloser LIMIT 1

If you do not know what columns are in your table, please execute the following command

    .schema Gloser

and post the results.

Simon.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: [EXTERNAL] Re: No unicode characters in sqlite3 console prg

larshgf
In reply to this post by Hick Gunter
Hi Hick Gunter,

This is the result:

sqlite> Select GRAESK, hex(GRAESK), DANSK from gloser;
taß???a ?|CF84CEB1CEB2CEADCF81CEBDCEB120CEB7|taverna
taß???a ?|CF84CEB1CEB2CEADCF81CEBDCEB120CEB7|taverna
taß???a ?|CF84CEB1CEB2CEADCF81CEBDCEB120CEB7|taverna
taß???a ?|CF84CEB1CEB2CEADCF81CEBDCEB120CEB7|taverna
taß???a ?|CF84CEB1CEB2CEADCF81CEBDCEB120CEB7|taverna
taß???a ?|CF84CEB1CEB2CEADCF81CEBDCEB120CEB7|taverna
taß???a ?|CF84CEB1CEB2CEADCF81CEBDCEB120CEB7|taverna
de µ?? ?e?...|CEB4CEB520CEBCCEBFCF8520CEBBCEB5CF822E2E2E|sig mig engang. (egl "du fortæller mig ikke")
d?µ?s?????f?? ?/?|CEB4CEB7CEBCCEBFCF83CEB9CEBFCEB3CF81CEACCF86CEBFCF8220CEBF2FCEB7|journalist
???µa t?|CEBACEADCF81CEBCCEB120CF84CEBF|mønt, småmønt
??p??a ?|CEBACEBFCF80CEADCEBBCEB120CEB7|pige
µa??t??a ?|CEBCCEB1CEB8CEAECF84CF81CEB9CEB120CEB7|elev, skoleelev (kvindelig)
µa??t?? ?|CEBCCEB1CEB8CEB7CF84CEAECF8220CEBF|elev, skoleelev (mandlig)
µp?????? t?|CEBCCF80CEBFCF85CEBACEACCEBBCEB920CF84CEBF|flaske
?p??|CF8CCF80CF89CF82|(konj) som, ligesom, sådan som; eftersom
p?t??? t?|CF80CEBFCF84CEAECF81CEB920CF84CEBF|glas (Både glas og flaske er intetkøn som vin. Øl er hunkøn)
p.?.= pa?ade??µat?? ????|CF802ECF872E3D20CF80CEB1CF81CEB1CEB4CEB5CEAFCEB3CEBCCEB1CF84CEBFCF8220CF87CEACCF81CEB7|f.eks. (for eksempel)
taß???a ?|CF84CEB1CEB2CEADCF81CEBDCEB120CEB7|taverna
taß???a ?|CF84CEB1CEB2CEADCF81CEBDCEB120CEB7|taverna
de µ?? ?e?...|CEB4CEB520CEBCCEBFCF8520CEBBCEB5CF822E2E2E|sig mig engang. (egl "du fortæller mig ikke")
d?µ?s?????f?? ?/?|CEB4CEB7CEBCCEBFCF83CEB9CEBFCEB3CF81CEACCF86CEBFCF8220CEBF2FCEB7|journalist
???µa t?|CEBACEADCF81CEBCCEB120CF84CEBF|mønt, småmønt
??p??a ?|CEBACEBFCF80CEADCEBBCEB120CEB7|pige
µa??t??a ?|CEBCCEB1CEB8CEAECF84CF81CEB9CEB120CEB7|elev, skoleelev (kvindelig)
µa??t?? ?|CEBCCEB1CEB8CEB7CF84CEAECF8220CEBF|elev, skoleelev (mandlig)
µp?????? t?|CEBCCF80CEBFCF85CEBACEACCEBBCEB920CF84CEBF|flaske
?p??|CF8CCF80CF89CF82|(konj) som, ligesom, sådan som; eftersom
p?t??? t?|CF80CEBFCF84CEAECF81CEB920CF84CEBF|glas (Både glas og flaske er intetkøn som vin. Øl er hunkøn)
p.?.= pa?ade??µat?? ????|CF802ECF872E3D20CF80CEB1CF81CEB1CEB4CEB5CEAFCEB3CEBCCEB1CF84CEBFCF8220CF87CEACCF81CEB7|f.eks. (for eksempel)
a???|CEB1CEBBCEBBCEAC|men
a???|CEB1CEBBCEBBCEAC|men
a???|CEB1CEBBCEBBCEAC|men
a???|CEB1CEBBCEBBCEAC|men
a???|CEB1CEBBCEBBCEAC|men
a???|CEB1CEBBCEBBCEAC|men
a???|CEB1CEBBCEBBCEAC|men
a???|CEB1CEBBCEBBCEAC|men
sqlite>

Does it tell you whats wrong?
/Lars


-----Oprindelig meddelelse-----
Fra: sqlite-users [mailto:[hidden email]] På vegne af Hick Gunter
Sendt: 13. august 2018 14:34
Til: 'SQLite mailing list'
Emne: Re: [sqlite] [EXTERNAL] Re: No unicode characters in sqlite3 console prg

Select graesk, hex(graesk), dansk from gloser;

Check the hex output to see if it conforms to UTF specs; if so, then your console is expecting a different encoding (probably ISO). If not, then your data was probably entered from a source the encodes in ISO.

-----Ursprüngliche Nachricht-----
Von: sqlite-users [mailto:[hidden email]] Im Auftrag von Lars Frederiksen
Gesendet: Montag, 13. August 2018 14:24
An: 'SQLite mailing list' <[hidden email]>
Betreff: Re: [sqlite] [EXTERNAL] Re: No unicode characters in sqlite3 console prg

Hi Hick Gunter,
I do not know exactly what you mean by this (hex(<textfield>))?
/Lars

-----Oprindelig meddelelse-----
Fra: sqlite-users [mailto:[hidden email]] På vegne af Hick Gunter
Sendt: 13. august 2018 14:07
Til: 'SQLite mailing list'
Emne: Re: [sqlite] [EXTERNAL] Re: No unicode characters in sqlite3 console prg

You need to show hex(<textfield>) before anyone can verify what encoding is stored.

-----Ursprüngliche Nachricht-----
Von: sqlite-users [mailto:[hidden email]] Im Auftrag von Lars Frederiksen
Gesendet: Montag, 13. August 2018 14:03
An: 'SQLite mailing list' <[hidden email]>
Betreff: [EXTERNAL] Re: [sqlite] No unicode characters in sqlite3 console prg

I use versio 3.19

Here is a print from cmd:

SQLite version 3.19.3 2017-06-08 14:26:16 Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> .open flashcard.db
sqlite> select * from Gloser;
2|taß???a ?|taverna|[01_16]
3|taß???a ?|taverna|[01_16]
4|taß???a ?|taverna|[01_16]
5|taß???a ?|taverna|[01_16]
16|taß???a ?|taverna|[01_16]
17|taß???a ?|taverna|[01_16]
18|taß???a ?|taverna|[01_16]
19|de µ?? ?e?...|sig mig engang. (egl "du fortæller mig ikke")|[01_08]
20|d?µ?s?????f?? ?/?|journalist|[01_08]
21|???µa t?|mønt, småmønt|[01_08]
22|??p??a ?|pige|[01_08]
23|µa??t??a ?|elev, skoleelev (kvindelig)|[01_08]
24|µa??t?? ?|elev, skoleelev (mandlig)|[01_08]
25|µp?????? t?|flaske|[01_08]
26|?p??|(konj) som, ligesom, sådan som; eftersom|[01_08]
27|p?t??? t?|glas (Både glas og flaske er intetkøn som vin. Øl er
27|hunkøn)|[01_08]
28|p.?.= pa?ade??µat?? ????|f.eks. (for eksempel)|[01_08]
29|taß???a ?|taverna|[01_16]
30|taß???a ?|taverna|[01_16]
31|de µ?? ?e?...|sig mig engang. (egl "du fortæller mig ikke")|[01_08]
32|d?µ?s?????f?? ?/?|journalist|[01_08]
33|???µa t?|mønt, småmønt|[01_08]
34|??p??a ?|pige|[01_08]
35|µa??t??a ?|elev, skoleelev (kvindelig)|[01_08]
36|µa??t?? ?|elev, skoleelev (mandlig)|[01_08]
37|µp?????? t?|flaske|[01_08]
38|?p??|(konj) som, ligesom, sådan som; eftersom|[01_08]
39|p?t??? t?|glas (Både glas og flaske er intetkøn som vin. Øl er
39|hunkøn)|[01_08]
40|p.?.= pa?ade??µat?? ????|f.eks. (for eksempel)|[01_08]
41|a???|men|[01_10]
42|a???|men|[01_10]
43|a???|men|[01_10]
44|a???|men|[01_10]
45|a???|men|[01_10]
46|a???|men|[01_10]
47|a???|men|[01_10]
48|a???|men|[01_10]

The second field should show greek words but only the letters that are identical in danish and greek is shown. The pure greek letters like β, γ, δ, ε, η etc are shown as questionmarks.
Do you want me to send you the database?

Best Regards
Lars

-----Oprindelig meddelelse-----
Fra: sqlite-users [mailto:[hidden email]] På vegne af Rowan Worth
Sendt: 13. august 2018 03:21
Til: SQLite mailing list
Emne: Re: [sqlite] No unicode characters in sqlite3 console prg

On 12 August 2018 at 00:51, Lars Frederiksen <[hidden email]> wrote:

> Is this an error of my windows 10 cmd prompt or is it a general
> problem that the sqlite3 console is not able to show unicode in the
> cmd-window.??
>
> No problem writing to the database and show (greek) unicode characters
> in the cmd prompt as I key them, but when I want to show the data with
> the SQL SELECT * FROM Gloser then the greek word is shown with most of
> its letters as questionmarks.
>

What version of sqlite are you using? There was significant efforts to improve windows console unicode handling awhile back, which I think made it in for 3.12/3.13.

If you're still seeing issues with a recent sqlite can you provide a short example which reproduces the issue?
-Rowan
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


___________________________________________
 Gunter Hick | Software Engineer | Scientific Games International GmbH | Klitschgasse 2-4, A-1130 Vienna | FN 157284 a, HG Wien, DVR: 0430013 | (O) +43 1 80100 - 0

May be privileged. May be confidential. Please delete if not the addressee.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


___________________________________________
 Gunter Hick | Software Engineer | Scientific Games International GmbH | Klitschgasse 2-4, A-1130 Vienna | FN 157284 a, HG Wien, DVR: 0430013 | (O) +43 1 80100 - 0

May be privileged. May be confidential. Please delete if not the addressee.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: [EXTERNAL] Re: No unicode characters in sqlite3 console prg

Hick Gunter
It tells me (courtesy of findencoding.com) that the contents is encoded in one of

- utf-8 (most probable)
- iso-8859-10
- iso-8859-14
- iso-8859-16
- iso-20222-cn
- iso-2022-cn-ext

While your console is probably expecting either a windows or an incompatible ISO encoding.

-----Ursprüngliche Nachricht-----
Von: sqlite-users [mailto:[hidden email]] Im Auftrag von Lars Frederiksen
Gesendet: Montag, 13. August 2018 14:46
An: 'SQLite mailing list' <[hidden email]>
Betreff: Re: [sqlite] [EXTERNAL] Re: No unicode characters in sqlite3 console prg

Hi Hick Gunter,

This is the result:

sqlite> Select GRAESK, hex(GRAESK), DANSK from gloser;
taß???a ?|CF84CEB1CEB2CEADCF81CEBDCEB120CEB7|taverna
taß???a ?|CF84CEB1CEB2CEADCF81CEBDCEB120CEB7|taverna
taß???a ?|CF84CEB1CEB2CEADCF81CEBDCEB120CEB7|taverna
taß???a ?|CF84CEB1CEB2CEADCF81CEBDCEB120CEB7|taverna
taß???a ?|CF84CEB1CEB2CEADCF81CEBDCEB120CEB7|taverna
taß???a ?|CF84CEB1CEB2CEADCF81CEBDCEB120CEB7|taverna
taß???a ?|CF84CEB1CEB2CEADCF81CEBDCEB120CEB7|taverna
de µ?? ?e?...|CEB4CEB520CEBCCEBFCF8520CEBBCEB5CF822E2E2E|sig mig engang. (egl "du fortæller mig ikke") d?µ?s?????f?? ?/?|CEB4CEB7CEBCCEBFCF83CEB9CEBFCEB3CF81CEACCF86CEBFCF8220CEBF2FCEB7|journalist
???µa t?|CEBACEADCF81CEBCCEB120CF84CEBF|mønt, småmønt ??p??a ?|CEBACEBFCF80CEADCEBBCEB120CEB7|pige
µa??t??a ?|CEBCCEB1CEB8CEAECF84CF81CEB9CEB120CEB7|elev, skoleelev (kvindelig) µa??t?? ?|CEBCCEB1CEB8CEB7CF84CEAECF8220CEBF|elev, skoleelev (mandlig) µp?????? t?|CEBCCF80CEBFCF85CEBACEACCEBBCEB920CF84CEBF|flaske
?p??|CF8CCF80CF89CF82|(konj) som, ligesom, sådan som; eftersom p?t??? t?|CF80CEBFCF84CEAECF81CEB920CF84CEBF|glas (Både glas og flaske er intetkøn som vin. Øl er hunkøn) p.?.= pa?ade??µat?? ????|CF802ECF872E3D20CF80CEB1CF81CEB1CEB4CEB5CEAFCEB3CEBCCEB1CF84CEBFCF8220CF87CEACCF81CEB7|f.eks. (for eksempel) taß???a ?|CF84CEB1CEB2CEADCF81CEBDCEB120CEB7|taverna
taß???a ?|CF84CEB1CEB2CEADCF81CEBDCEB120CEB7|taverna
de µ?? ?e?...|CEB4CEB520CEBCCEBFCF8520CEBBCEB5CF822E2E2E|sig mig engang. (egl "du fortæller mig ikke") d?µ?s?????f?? ?/?|CEB4CEB7CEBCCEBFCF83CEB9CEBFCEB3CF81CEACCF86CEBFCF8220CEBF2FCEB7|journalist
???µa t?|CEBACEADCF81CEBCCEB120CF84CEBF|mønt, småmønt ??p??a ?|CEBACEBFCF80CEADCEBBCEB120CEB7|pige
µa??t??a ?|CEBCCEB1CEB8CEAECF84CF81CEB9CEB120CEB7|elev, skoleelev (kvindelig) µa??t?? ?|CEBCCEB1CEB8CEB7CF84CEAECF8220CEBF|elev, skoleelev (mandlig) µp?????? t?|CEBCCF80CEBFCF85CEBACEACCEBBCEB920CF84CEBF|flaske
?p??|CF8CCF80CF89CF82|(konj) som, ligesom, sådan som; eftersom p?t??? t?|CF80CEBFCF84CEAECF81CEB920CF84CEBF|glas (Både glas og flaske er intetkøn som vin. Øl er hunkøn) p.?.= pa?ade??µat?? ????|CF802ECF872E3D20CF80CEB1CF81CEB1CEB4CEB5CEAFCEB3CEBCCEB1CF84CEBFCF8220CF87CEACCF81CEB7|f.eks. (for eksempel) a???|CEB1CEBBCEBBCEAC|men a???|CEB1CEBBCEBBCEAC|men a???|CEB1CEBBCEBBCEAC|men a???|CEB1CEBBCEBBCEAC|men a???|CEB1CEBBCEBBCEAC|men a???|CEB1CEBBCEBBCEAC|men a???|CEB1CEBBCEBBCEAC|men a???|CEB1CEBBCEBBCEAC|men
sqlite>

Does it tell you whats wrong?
/Lars


-----Oprindelig meddelelse-----
Fra: sqlite-users [mailto:[hidden email]] På vegne af Hick Gunter
Sendt: 13. august 2018 14:34
Til: 'SQLite mailing list'
Emne: Re: [sqlite] [EXTERNAL] Re: No unicode characters in sqlite3 console prg

Select graesk, hex(graesk), dansk from gloser;

Check the hex output to see if it conforms to UTF specs; if so, then your console is expecting a different encoding (probably ISO). If not, then your data was probably entered from a source the encodes in ISO.

-----Ursprüngliche Nachricht-----
Von: sqlite-users [mailto:[hidden email]] Im Auftrag von Lars Frederiksen
Gesendet: Montag, 13. August 2018 14:24
An: 'SQLite mailing list' <[hidden email]>
Betreff: Re: [sqlite] [EXTERNAL] Re: No unicode characters in sqlite3 console prg

Hi Hick Gunter,
I do not know exactly what you mean by this (hex(<textfield>))?
/Lars

-----Oprindelig meddelelse-----
Fra: sqlite-users [mailto:[hidden email]] På vegne af Hick Gunter
Sendt: 13. august 2018 14:07
Til: 'SQLite mailing list'
Emne: Re: [sqlite] [EXTERNAL] Re: No unicode characters in sqlite3 console prg

You need to show hex(<textfield>) before anyone can verify what encoding is stored.

-----Ursprüngliche Nachricht-----
Von: sqlite-users [mailto:[hidden email]] Im Auftrag von Lars Frederiksen
Gesendet: Montag, 13. August 2018 14:03
An: 'SQLite mailing list' <[hidden email]>
Betreff: [EXTERNAL] Re: [sqlite] No unicode characters in sqlite3 console prg

I use versio 3.19

Here is a print from cmd:

SQLite version 3.19.3 2017-06-08 14:26:16 Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> .open flashcard.db
sqlite> select * from Gloser;
2|taß???a ?|taverna|[01_16]
3|taß???a ?|taverna|[01_16]
4|taß???a ?|taverna|[01_16]
5|taß???a ?|taverna|[01_16]
16|taß???a ?|taverna|[01_16]
17|taß???a ?|taverna|[01_16]
18|taß???a ?|taverna|[01_16]
19|de µ?? ?e?...|sig mig engang. (egl "du fortæller mig ikke")|[01_08]
20|d?µ?s?????f?? ?/?|journalist|[01_08]
21|???µa t?|mønt, småmønt|[01_08]
22|??p??a ?|pige|[01_08]
23|µa??t??a ?|elev, skoleelev (kvindelig)|[01_08]
24|µa??t?? ?|elev, skoleelev (mandlig)|[01_08]
25|µp?????? t?|flaske|[01_08]
26|?p??|(konj) som, ligesom, sådan som; eftersom|[01_08]
27|p?t??? t?|glas (Både glas og flaske er intetkøn som vin. Øl er
27|hunkøn)|[01_08]
28|p.?.= pa?ade??µat?? ????|f.eks. (for eksempel)|[01_08]
29|taß???a ?|taverna|[01_16]
30|taß???a ?|taverna|[01_16]
31|de µ?? ?e?...|sig mig engang. (egl "du fortæller mig ikke")|[01_08]
32|d?µ?s?????f?? ?/?|journalist|[01_08]
33|???µa t?|mønt, småmønt|[01_08]
34|??p??a ?|pige|[01_08]
35|µa??t??a ?|elev, skoleelev (kvindelig)|[01_08]
36|µa??t?? ?|elev, skoleelev (mandlig)|[01_08]
37|µp?????? t?|flaske|[01_08]
38|?p??|(konj) som, ligesom, sådan som; eftersom|[01_08]
39|p?t??? t?|glas (Både glas og flaske er intetkøn som vin. Øl er
39|hunkøn)|[01_08]
40|p.?.= pa?ade??µat?? ????|f.eks. (for eksempel)|[01_08]
41|a???|men|[01_10]
42|a???|men|[01_10]
43|a???|men|[01_10]
44|a???|men|[01_10]
45|a???|men|[01_10]
46|a???|men|[01_10]
47|a???|men|[01_10]
48|a???|men|[01_10]

The second field should show greek words but only the letters that are identical in danish and greek is shown. The pure greek letters like β, γ, δ, ε, η etc are shown as questionmarks.
Do you want me to send you the database?

Best Regards
Lars

-----Oprindelig meddelelse-----
Fra: sqlite-users [mailto:[hidden email]] På vegne af Rowan Worth
Sendt: 13. august 2018 03:21
Til: SQLite mailing list
Emne: Re: [sqlite] No unicode characters in sqlite3 console prg

On 12 August 2018 at 00:51, Lars Frederiksen <[hidden email]> wrote:

> Is this an error of my windows 10 cmd prompt or is it a general
> problem that the sqlite3 console is not able to show unicode in the
> cmd-window.??
>
> No problem writing to the database and show (greek) unicode characters
> in the cmd prompt as I key them, but when I want to show the data with
> the SQL SELECT * FROM Gloser then the greek word is shown with most of
> its letters as questionmarks.
>

What version of sqlite are you using? There was significant efforts to improve windows console unicode handling awhile back, which I think made it in for 3.12/3.13.

If you're still seeing issues with a recent sqlite can you provide a short example which reproduces the issue?
-Rowan
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


___________________________________________
 Gunter Hick | Software Engineer | Scientific Games International GmbH | Klitschgasse 2-4, A-1130 Vienna | FN 157284 a, HG Wien, DVR: 0430013 | (O) +43 1 80100 - 0

May be privileged. May be confidential. Please delete if not the addressee.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


___________________________________________
 Gunter Hick | Software Engineer | Scientific Games International GmbH | Klitschgasse 2-4, A-1130 Vienna | FN 157284 a, HG Wien, DVR: 0430013 | (O) +43 1 80100 - 0

May be privileged. May be confidential. Please delete if not the addressee.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


___________________________________________
 Gunter Hick | Software Engineer | Scientific Games International GmbH | Klitschgasse 2-4, A-1130 Vienna | FN 157284 a, HG Wien, DVR: 0430013 | (O) +43 1 80100 - 0

May be privileged. May be confidential. Please delete if not the addressee.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: No unicode characters in sqlite3 console prg

Olivier Mascia
In reply to this post by larshgf
> Le 13 août 2018 à 14:22, Lars Frederiksen <[hidden email]> a écrit :
>
> I use Windows 10 and have tried
> chcp 65001
> without any luck. Still the same questionmarks
> /Lars

Of course. :) That is Windows console.  You also have to elect to use an alternate font with your console, one that encompasses enough unicode glyphs for your needs.  Click the upper left corner of your console window, choose properties, then font.  You might have to find additional fonts and install them.

Consider this workflow using a french speaking Windows 10:


I:\>chcp
Page de codes active : 850

I:\>sqlite3 école.db "create table école (élève text); insert into école values('élisabeth');"

Comment: While my code page is 850, I type some extended characters to create a file, a table, a column and a value all using at least one extended character which is properly supported by my code page 850. The input to SQLite is correctly interpreted (see proof later on). But I would not have been able to enter any greek characters or arabic, not part of my code page 850.

I:\>sqlite3 école.db
SQLite version 3.24.0 2018-06-04 19:24:41
Enter ".help" for usage hints.
sqlite> .once test.sql
sqlite> .dump
sqlite> .q

I:\>type test.sql
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
CREATE TABLE ├®cole (├®l├¿ve text);
INSERT INTO "├®cole" VALUES('├®lisabeth');
COMMIT;

Comment: So when dumping to file, there SQLite properly outputs UTF-8 without my code page settings getting in the way. That is nice and should never ever be changed. The fact that it outputted UTF-8 can clearly be seen above because the various extended characters, encoded as UTF8 appear as individual bytes mapped to my default code page 850.

Now switch the console code page to UTF-8 (65001):

I:\>chcp 65001
Page de codes active : 65001

I:\>type test.sql
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
CREATE TABLE école (élève text);
INSERT INTO "école" VALUES('élisabeth');
COMMIT;

And the text file content appears OK, confirming the actuel text file data is UTF-8 (which you could validate by other ways, using a text editor properly supporting the encoding).

--
Best Regards, Meilleures salutations, Met vriendelijke groeten,
Olivier Mascia


_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: [EXTERNAL] Re: No unicode characters in sqlite3 console prg

larshgf
In reply to this post by Hick Gunter
Hi everybody,

And thank you very much for your help.
The problem is not a BIG issue for me because i primarily use the console program for creating databases. The unicode chrs are even inserted correct in the greek field, because I can read them correctly in my DBGrid of my Delphi program.
I just think it is an odd "behavior" of the cmd, but never mind. I might try to experiment a little with codepages etc and see if this helps.
Thank you!
Lars

-----Oprindelig meddelelse-----
Fra: sqlite-users [mailto:[hidden email]] På vegne af Hick Gunter
Sendt: 13. august 2018 15:14
Til: 'SQLite mailing list'
Emne: Re: [sqlite] [EXTERNAL] Re: No unicode characters in sqlite3 console prg

It tells me (courtesy of findencoding.com) that the contents is encoded in one of

- utf-8 (most probable)
- iso-8859-10
- iso-8859-14
- iso-8859-16
- iso-20222-cn
- iso-2022-cn-ext

While your console is probably expecting either a windows or an incompatible ISO encoding.

-----Ursprüngliche Nachricht-----
Von: sqlite-users [mailto:[hidden email]] Im Auftrag von Lars Frederiksen
Gesendet: Montag, 13. August 2018 14:46
An: 'SQLite mailing list' <[hidden email]>
Betreff: Re: [sqlite] [EXTERNAL] Re: No unicode characters in sqlite3 console prg

Hi Hick Gunter,

This is the result:

sqlite> Select GRAESK, hex(GRAESK), DANSK from gloser;
taß???a ?|CF84CEB1CEB2CEADCF81CEBDCEB120CEB7|taverna
taß???a ?|CF84CEB1CEB2CEADCF81CEBDCEB120CEB7|taverna
taß???a ?|CF84CEB1CEB2CEADCF81CEBDCEB120CEB7|taverna
taß???a ?|CF84CEB1CEB2CEADCF81CEBDCEB120CEB7|taverna
taß???a ?|CF84CEB1CEB2CEADCF81CEBDCEB120CEB7|taverna
taß???a ?|CF84CEB1CEB2CEADCF81CEBDCEB120CEB7|taverna
taß???a ?|CF84CEB1CEB2CEADCF81CEBDCEB120CEB7|taverna
de µ?? ?e?...|CEB4CEB520CEBCCEBFCF8520CEBBCEB5CF822E2E2E|sig mig engang. (egl "du fortæller mig ikke") d?µ?s?????f?? ?/?|CEB4CEB7CEBCCEBFCF83CEB9CEBFCEB3CF81CEACCF86CEBFCF8220CEBF2FCEB7|journalist
???µa t?|CEBACEADCF81CEBCCEB120CF84CEBF|mønt, småmønt ??p??a ?|CEBACEBFCF80CEADCEBBCEB120CEB7|pige
µa??t??a ?|CEBCCEB1CEB8CEAECF84CF81CEB9CEB120CEB7|elev, skoleelev (kvindelig) µa??t?? ?|CEBCCEB1CEB8CEB7CF84CEAECF8220CEBF|elev, skoleelev (mandlig) µp?????? t?|CEBCCF80CEBFCF85CEBACEACCEBBCEB920CF84CEBF|flaske
?p??|CF8CCF80CF89CF82|(konj) som, ligesom, sådan som; eftersom p?t??? t?|CF80CEBFCF84CEAECF81CEB920CF84CEBF|glas (Både glas og flaske er intetkøn som vin. Øl er hunkøn) p.?.= pa?ade??µat?? ????|CF802ECF872E3D20CF80CEB1CF81CEB1CEB4CEB5CEAFCEB3CEBCCEB1CF84CEBFCF8220CF87CEACCF81CEB7|f.eks. (for eksempel) taß???a ?|CF84CEB1CEB2CEADCF81CEBDCEB120CEB7|taverna
taß???a ?|CF84CEB1CEB2CEADCF81CEBDCEB120CEB7|taverna
de µ?? ?e?...|CEB4CEB520CEBCCEBFCF8520CEBBCEB5CF822E2E2E|sig mig engang. (egl "du fortæller mig ikke") d?µ?s?????f?? ?/?|CEB4CEB7CEBCCEBFCF83CEB9CEBFCEB3CF81CEACCF86CEBFCF8220CEBF2FCEB7|journalist
???µa t?|CEBACEADCF81CEBCCEB120CF84CEBF|mønt, småmønt ??p??a ?|CEBACEBFCF80CEADCEBBCEB120CEB7|pige
µa??t??a ?|CEBCCEB1CEB8CEAECF84CF81CEB9CEB120CEB7|elev, skoleelev (kvindelig) µa??t?? ?|CEBCCEB1CEB8CEB7CF84CEAECF8220CEBF|elev, skoleelev (mandlig) µp?????? t?|CEBCCF80CEBFCF85CEBACEACCEBBCEB920CF84CEBF|flaske
?p??|CF8CCF80CF89CF82|(konj) som, ligesom, sådan som; eftersom p?t??? t?|CF80CEBFCF84CEAECF81CEB920CF84CEBF|glas (Både glas og flaske er intetkøn som vin. Øl er hunkøn) p.?.= pa?ade??µat?? ????|CF802ECF872E3D20CF80CEB1CF81CEB1CEB4CEB5CEAFCEB3CEBCCEB1CF84CEBFCF8220CF87CEACCF81CEB7|f.eks. (for eksempel) a???|CEB1CEBBCEBBCEAC|men a???|CEB1CEBBCEBBCEAC|men a???|CEB1CEBBCEBBCEAC|men a???|CEB1CEBBCEBBCEAC|men a???|CEB1CEBBCEBBCEAC|men a???|CEB1CEBBCEBBCEAC|men a???|CEB1CEBBCEBBCEAC|men a???|CEB1CEBBCEBBCEAC|men
sqlite>

Does it tell you whats wrong?
/Lars


-----Oprindelig meddelelse-----
Fra: sqlite-users [mailto:[hidden email]] På vegne af Hick Gunter
Sendt: 13. august 2018 14:34
Til: 'SQLite mailing list'
Emne: Re: [sqlite] [EXTERNAL] Re: No unicode characters in sqlite3 console prg

Select graesk, hex(graesk), dansk from gloser;

Check the hex output to see if it conforms to UTF specs; if so, then your console is expecting a different encoding (probably ISO). If not, then your data was probably entered from a source the encodes in ISO.

-----Ursprüngliche Nachricht-----
Von: sqlite-users [mailto:[hidden email]] Im Auftrag von Lars Frederiksen
Gesendet: Montag, 13. August 2018 14:24
An: 'SQLite mailing list' <[hidden email]>
Betreff: Re: [sqlite] [EXTERNAL] Re: No unicode characters in sqlite3 console prg

Hi Hick Gunter,
I do not know exactly what you mean by this (hex(<textfield>))?
/Lars

-----Oprindelig meddelelse-----
Fra: sqlite-users [mailto:[hidden email]] På vegne af Hick Gunter
Sendt: 13. august 2018 14:07
Til: 'SQLite mailing list'
Emne: Re: [sqlite] [EXTERNAL] Re: No unicode characters in sqlite3 console prg

You need to show hex(<textfield>) before anyone can verify what encoding is stored.

-----Ursprüngliche Nachricht-----
Von: sqlite-users [mailto:[hidden email]] Im Auftrag von Lars Frederiksen
Gesendet: Montag, 13. August 2018 14:03
An: 'SQLite mailing list' <[hidden email]>
Betreff: [EXTERNAL] Re: [sqlite] No unicode characters in sqlite3 console prg

I use versio 3.19

Here is a print from cmd:

SQLite version 3.19.3 2017-06-08 14:26:16 Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> .open flashcard.db
sqlite> select * from Gloser;
2|taß???a ?|taverna|[01_16]
3|taß???a ?|taverna|[01_16]
4|taß???a ?|taverna|[01_16]
5|taß???a ?|taverna|[01_16]
16|taß???a ?|taverna|[01_16]
17|taß???a ?|taverna|[01_16]
18|taß???a ?|taverna|[01_16]
19|de µ?? ?e?...|sig mig engang. (egl "du fortæller mig ikke")|[01_08]
20|d?µ?s?????f?? ?/?|journalist|[01_08]
21|???µa t?|mønt, småmønt|[01_08]
22|??p??a ?|pige|[01_08]
23|µa??t??a ?|elev, skoleelev (kvindelig)|[01_08]
24|µa??t?? ?|elev, skoleelev (mandlig)|[01_08]
25|µp?????? t?|flaske|[01_08]
26|?p??|(konj) som, ligesom, sådan som; eftersom|[01_08]
27|p?t??? t?|glas (Både glas og flaske er intetkøn som vin. Øl er
27|hunkøn)|[01_08]
28|p.?.= pa?ade??µat?? ????|f.eks. (for eksempel)|[01_08]
29|taß???a ?|taverna|[01_16]
30|taß???a ?|taverna|[01_16]
31|de µ?? ?e?...|sig mig engang. (egl "du fortæller mig ikke")|[01_08]
32|d?µ?s?????f?? ?/?|journalist|[01_08]
33|???µa t?|mønt, småmønt|[01_08]
34|??p??a ?|pige|[01_08]
35|µa??t??a ?|elev, skoleelev (kvindelig)|[01_08]
36|µa??t?? ?|elev, skoleelev (mandlig)|[01_08]
37|µp?????? t?|flaske|[01_08]
38|?p??|(konj) som, ligesom, sådan som; eftersom|[01_08]
39|p?t??? t?|glas (Både glas og flaske er intetkøn som vin. Øl er
39|hunkøn)|[01_08]
40|p.?.= pa?ade??µat?? ????|f.eks. (for eksempel)|[01_08]
41|a???|men|[01_10]
42|a???|men|[01_10]
43|a???|men|[01_10]
44|a???|men|[01_10]
45|a???|men|[01_10]
46|a???|men|[01_10]
47|a???|men|[01_10]
48|a???|men|[01_10]

The second field should show greek words but only the letters that are identical in danish and greek is shown. The pure greek letters like β, γ, δ, ε, η etc are shown as questionmarks.
Do you want me to send you the database?

Best Regards
Lars

-----Oprindelig meddelelse-----
Fra: sqlite-users [mailto:[hidden email]] På vegne af Rowan Worth
Sendt: 13. august 2018 03:21
Til: SQLite mailing list
Emne: Re: [sqlite] No unicode characters in sqlite3 console prg

On 12 August 2018 at 00:51, Lars Frederiksen <[hidden email]> wrote:

> Is this an error of my windows 10 cmd prompt or is it a general
> problem that the sqlite3 console is not able to show unicode in the
> cmd-window.??
>
> No problem writing to the database and show (greek) unicode characters
> in the cmd prompt as I key them, but when I want to show the data with
> the SQL SELECT * FROM Gloser then the greek word is shown with most of
> its letters as questionmarks.
>

What version of sqlite are you using? There was significant efforts to improve windows console unicode handling awhile back, which I think made it in for 3.12/3.13.

If you're still seeing issues with a recent sqlite can you provide a short example which reproduces the issue?
-Rowan
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


___________________________________________
 Gunter Hick | Software Engineer | Scientific Games International GmbH | Klitschgasse 2-4, A-1130 Vienna | FN 157284 a, HG Wien, DVR: 0430013 | (O) +43 1 80100 - 0

May be privileged. May be confidential. Please delete if not the addressee.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


___________________________________________
 Gunter Hick | Software Engineer | Scientific Games International GmbH | Klitschgasse 2-4, A-1130 Vienna | FN 157284 a, HG Wien, DVR: 0430013 | (O) +43 1 80100 - 0

May be privileged. May be confidential. Please delete if not the addressee.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


___________________________________________
 Gunter Hick | Software Engineer | Scientific Games International GmbH | Klitschgasse 2-4, A-1130 Vienna | FN 157284 a, HG Wien, DVR: 0430013 | (O) +43 1 80100 - 0

May be privileged. May be confidential. Please delete if not the addressee.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: [EXTERNAL] Re: No unicode characters in sqlite3 console prg

Simon Slavin-3
In reply to this post by larshgf
On 13 Aug 2018, at 1:45pm, Lars Frederiksen <[hidden email]> wrote:

> This is the result:
>
> sqlite> Select GRAESK, hex(GRAESK), DANSK from gloser;
> taß???a ?|CF84CEB1CEB2CEADCF81CEBDCEB120CEB7|taverna

"taverna" in greek should be "ταβέρνα".  In UTF-8 I get

CF84 CEB1 CEB2 CEAD CF81 CEBD CEB1

Examining what you show ...

CF84 CEB1 CEB2 CEAD CF81 CEBD CEB1 20 CEB7

Which is the correct string with, for some reason, " η" on the end.  So the encoding of your database should be UTF-8.

It is possible that the contents of your database do not agree with the stated encoding of the database.  Please execute the command

   PRAGMA encoding;

in the shell tool and tell us what the result is.  For details see

<https://www.sqlite.org/pragma.html#pragma_encoding>

if you get UTF-8, then the fault is entirely within Windows, and the database contains exactly what you would want.

Simon.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Reply | Threaded
Open this post in threaded view
|

Re: [EXTERNAL] Re: No unicode characters in sqlite3 console prg

larshgf
Hi Simon,

PRAGMA Encoding gave me "utf-8" and the db gives me correct greek letters in my DBgrid in my Delphi program. So that is ok!
BTW: ταβέρνα η - the last letter represents the definite article :-)
I will read more about PRAGMA - here I am absolutely blank. PRAGMA is also used in the FireDAC components of my Delphi programming tool
Thank you very much.

/Lars


-----Oprindelig meddelelse-----
Fra: sqlite-users [mailto:[hidden email]] På vegne af Simon Slavin
Sendt: 13. august 2018 16:24
Til: SQLite mailing list
Emne: Re: [sqlite] [EXTERNAL] Re: No unicode characters in sqlite3 console prg

On 13 Aug 2018, at 1:45pm, Lars Frederiksen <[hidden email]> wrote:

> This is the result:
>
> sqlite> Select GRAESK, hex(GRAESK), DANSK from gloser;
> taß???a ?|CF84CEB1CEB2CEADCF81CEBDCEB120CEB7|taverna

"taverna" in greek should be "ταβέρνα".  In UTF-8 I get

CF84 CEB1 CEB2 CEAD CF81 CEBD CEB1

Examining what you show ...

CF84 CEB1 CEB2 CEAD CF81 CEBD CEB1 20 CEB7

Which is the correct string with, for some reason, " η" on the end.  So the encoding of your database should be UTF-8.

It is possible that the contents of your database do not agree with the stated encoding of the database.  Please execute the command

   PRAGMA encoding;

in the shell tool and tell us what the result is.  For details see

<https://www.sqlite.org/pragma.html#pragma_encoding>

if you get UTF-8, then the fault is entirely within Windows, and the database contains exactly what you would want.

Simon.
_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users