I can't undestand an assertion in a Lemon Parser

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

I can't undestand an assertion in a Lemon Parser

NightClicer Parhom
(English is not my native language; please excuse typing errors.)

Greetings!

I have a fun with my own simple project Lemon Server. It is a simple HTTP Server where HTTP protocol (RFC7230) was implemented with a Lemon Parser. I got a result grammar file but a fresh revision of the Lemon Parser can not parse incoming requests.

Lemon Server repo: https://github.com/devAarno/lemon-server/tree/feature/LS-16-Lemon-update
Grammar file: https://github.com/devAarno/lemon-server/blob/feature/LS-16-Lemon-update/src/lemonHttp/http11.y

As I see in a Fossil repository, on 2017-12-27 18:19:06 a developer 'drh' has added a line "tool/lempar.c:514":

assert( i>=0 && i+YYNTOKEN<=sizeof(yy_lookahead)/sizeof(yy_lookahead[0]) );

(SHA3-256: 1b22b42e59793af19c69a2e5f6822883cc2687d4a0d9b9280bbff885276c6baa, https://www.sqlite.org/src/info/1b22b42e59793af1)

Last commit (drh on 2018-02-09 15:04:51) says that this line was split into:

assert( i>=0 );
assert( i+YYNTOKEN<=(int)sizeof(yy_lookahead)/sizeof(yy_lookahead[0]) );

(SHA3-256: a6c3115483d597fc77ab19fdcfd1d3437cad7e467081ad8c5315fb98c115eed9, https://www.sqlite.org/src/info/a6c3115483d597fc)

So, I have some problems with a second
assert (i+YYNTOKEN<=(int)sizeof(yy_lookahead)/sizeof(yy_lookahead[0])).

My tests say that parser works well without this assert.
For example, test "test_byRawRequest1" says:

(test in https://github.com/devAarno/lemon-server/blob/feature/LS-16-Lemon-update/tests/t_1.c)

parser >>Reduce 35 [http_version ::= H T T P SLASH ZERO|ONE|TWO|THREE|FOUR|FIVE|SIX|SEVEN|EIGHT|NINE DOT ZERO|ONE|TWO|THREE|FOUR|FIVE|SIX|SEVEN|EIGHT|NINE], go to state 19.
parser >>... then shift 'http_version', go to state 18
parser >>Shift 'CR', go to state 40
parser >>Return. Stack=[method SP request_target SP http_version CR]
parser >>Input 'CLF' in state 40
parser >>Shift 'CLF', pending reduce 54
parser >>Return. Stack=[method SP request_target SP http_version CR CLF]
parser >>Input 'CR' with pending reduce 54
parser >>Reduce 54 [crlf ::= CR CLF], go to state 18.
parser >>... then shift 'crlf', pending reduce 42
parser >>Reduce 42 [request_line ::= method SP request_target SP http_version crlf], go to state 0.
parser >>... then shift 'request_line', go to state 22
parser >>Reduce 48 [http_headers ::=].
parser >>... then shift 'http_headers', go to state 8
parser >>Shift 'CR', go to state 39
parser >>Return. Stack=[request_line http_headers CR]
parser >>Input 'CLF' in state 39
parser >>Shift 'CLF', pending reduce 40
parser >>Return. Stack=[request_line http_headers CR CLF]
parser >>Input 'CONTROL' with pending reduce 40
parser >>Reduce 40 [finalcrlf ::= CR CLF], go to state 8.
DONE
parser >>... then shift 'finalcrlf', go to state 42
parser >>WILDCARD CONTROL => ANY
parser >>Shift 'CONTROL', go to state 41
parser >>Return. Stack=[request_line http_headers finalcrlf CONTROL]
parser >>Input '$' in state 41
parser >>Reduce 41 [http_message ::= request_line http_headers finalcrlf ANY], go to state 0.
parser >>... then shift 'http_message', pending reduce -2
parser >>Accept!
/home/aarno/NetBeansProjects/lemon-server/tests/t_1.c:165:test_byRawRequest1:PASS

-----------------------
1 Tests 0 Failures 0 Ignored


If I uncomment this assert, I shall get:

parser >>Reduce 35 [http_version ::= H T T P SLASH ZERO|ONE|TWO|THREE|FOUR|FIVE|SIX|SEVEN|EIGHT|NINE DOT ZERO|ONE|TWO|THREE|FOUR|FIVE|SIX|SEVEN|EIGHT|NINE], go to state 19.
parser >>... then shift 'http_version', go to state 18
parser >>Shift 'CR', go to state 40
parser >>Return. Stack=[method SP request_target SP http_version CR]
parser >>Input 'CLF' in state 40
parser >>Shift 'CLF', pending reduce 54
parser >>Return. Stack=[method SP request_target SP http_version CR CLF]
parser >>Input 'CR' with pending reduce 54
parser >>Reduce 54 [crlf ::= CR CLF], go to state 18.
parser >>... then shift 'crlf', pending reduce 42
parser >>Reduce 42 [request_line ::= method SP request_target SP http_version crlf], go to state 0.
parser >>... then shift 'request_line', go to state 22
t_1: /home/aarno/NetBeansProjects/lemon-server/src/lemonHttp/http11.c:1245: yy_find_shift_action: Assertion `i+YYNTOKEN<=(int)sizeof(yy_lookahead)/sizeof(yy_lookahead[0])' failed.

gdb says:
(gdb) print i
$2 = 1019
(gdb) print YYNTOKEN
$3 = 104
(gdb) print sizeof(yy_lookahead)
$4 = 1390
(gdb) print sizeof(yy_lookahead[0])
$5 = 1

(Linux x86-64, Ubuntu 17.10, gcc (Ubuntu 7.2.0-8ubuntu3.2) 7.2.0)

I'm not an expert in a Lemon Parser's source code, so I would like to have some explanation about this assert. What is exactly it check?
What shall I do to avoid this assert? Is it bug in my code?

If you are interested to build my source (branch feature/LS-16-Lemon-update), please, follow:
1. "git clone -b feature/LS-16-Lemon-update https://github.com/devAarno/lemon-server.git"
2. "cd lemon-server/"
3. "cmake -DWITHTESTS=1 ." (git and cmake are required)
4. "make clean all" (use after ./src/lemonHttp/lempar.c changing)
5. "./tests/t_1"

Additional question:

how properly use "ParseHTTP11Init()" with "Parse_ENGINEALWAYSONSTACK" definition?
I use hack like

https://github.com/devAarno/lemon-server/blob/feature/LS-16-Lemon-update/src/lemonHttp/lempar.c:362
https://github.com/devAarno/lemon-server/blob/feature/LS-16-Lemon-update/src/lemonHttp/parser.c:135

Can you show me an example where Lemon Parser works with "Parse_ENGINEALWAYSONSTACK"
and unmodified "lempar.c"?

Thanks for your attention.

Sincerely, Stanislav.


_______________________________________________
sqlite-users mailing list
[hidden email]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users