Bugs item #3437225, was opened at 2011-11-12 23:44
Message generated for change (Comment added) made by jpff
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=564599&aid=3437225&group_id=81968

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: csound
Group: csound5
>Status: Closed
>Resolution: Accepted
Priority: 5
Private: No
Submitted By: Andres Cabrera (mantaraya36)
Assigned to: John ffitch (jpff)
Summary: new parser #include bug

Initial Comment:
Hi All, there is a bug with `#include' in the new parser (only for orc
file or CsInstruments section in the csd file). The last line of a
file included through `#include' is analyzed two times by the lexer.
For example:

;; foo.txt
prints "=== FIRST  LINE ===\n"
prints "=== SECOND LINE ===\n"
prints "=== LAST   LINE ===\n"


<CsoundSynthesizer>
<CsInstruments>
#include "foo.txt"
instr 1
endin
</CsInstruments>
<CsScore>
i1 0 0
</CsScore>
</CsoundSynthesizer>


Tested with Csound 5.14, git commit 78d3db0ff720c7feec3914a20850aeeb75c084ef

uname -a
Linux pantera 2.6.33.7.2-rt30 #2 SMP PREEMPT RT Mon Oct 24 12:33:25 CEST 2011
x86_64 AMD Phenom(tm) II X6 1090T Processor AuthenticAMD GNU/Linux

csound -n include_test.csd
...
********************
* USING NEW PARSER *
********************
Testing...
...
<ast>
<tree type="292" name="T_OPCODE0" opname0="prints" >
 <treeright type="321" name="T_STRCONST" str=""=== FIRST  LINE ===\n"" >
 </treeright>
</tree>
<treenext type="292" name="T_OPCODE0" opname0="prints" >
 <treeright type="321" name="T_STRCONST" str=""=== SECOND LINE ===\n"" >
 </treeright>
</treenext>
<treenext type="292" name="T_OPCODE0" opname0="prints" >
 <treeright type="321" name="T_STRCONST" str=""=== LAST   LINE ===\n"" >
 </treeright>
</treenext>
<treenext type="292" name="T_OPCODE0" opname0="prints" >
 <treeright type="321" name="T_STRCONST" str=""=== LAST   LINE ===\n"" >
 </treeright>
</treenext>
<treenext type="301" name="T_INSTR" >
 <treeleft type="340" name="T_INTGR" value="1" >
 </treeleft>
</treenext>
</ast>
...
=== FIRST  LINE ===
=== SECOND LINE ===
=== LAST   LINE ===
=== LAST   LINE ===
orch now loaded
...


The cause is `PARM->buffer' (or yyextra->buffer) in csound_orc.l. The
scanner works well in the buffer state "yypushed" in the `do_include'
function, but after the `yypop_buffer_state', in the <<EOF>> rule,
PARM->buffer is still 'prints "=== LAST LINE ===\n"' and PARM->lBuffer
is 31 (the lenght of PARM->buffer), so this buffered line will be
analyzed again (in `get_next_char' used by `YY_INPUT'). A temporary
solution is


diff -ur csound5~/Engine/csound_orc.l csound5/Engine/csound_orc.l
--- csound5~/Engine/csound_orc.l        2011-11-08 13:51:46.000000000 +0100
+++ csound5/Engine/csound_orc.l 2011-11-12 17:36:27.000000000 +0100
@@ -432,6 +432,8 @@
                  yypop_buffer_state(yyscanner);
                  if ( !YY_CURRENT_BUFFER ) yyterminate();
                  printf("End of input; popping to %p\n", YY_CURRENT_BUFFER);
+                  if (PARM->isInclude)
+                    PARM->lBuffer = PARM->isInclude = 0;
                  n = PARM->alt_stack[--PARM->macro_stack_ptr].n;
                  printf("n=%d\n", n);
                  if (n!=0) {
@@ -530,6 +532,7 @@
    }
    PARM->alt_stack[PARM->macro_stack_ptr].n = 0;
    PARM->alt_stack[PARM->macro_stack_ptr++].s = NULL;
+    PARM->isInclude = 1;
    yypush_buffer_state(yy_create_buffer( yyin, YY_BUF_SIZE, yyscanner), yyscanner);
 }

diff -ur csound5~/Engine/parse_param.h csound5/Engine/parse_param.h
--- csound5~/Engine/parse_param.h       2011-11-08 13:51:46.000000000 +0100
+++ csound5/Engine/parse_param.h        2011-11-12 17:36:45.000000000 +0100
@@ -30,6 +30,7 @@
    unsigned int macro_stack_ptr;
    char            *xstrbuff;
    int             xstrptr,xstrmax;
+    unsigned char   isInclude;
 } PARSE_PARM;

 #define lMaxBuffer (1000)

The `if' condition is necessary, otherwise the $NAME macro doesn't work.

tito

----------------------------------------------------------------------

>Comment By: John ffitch (jpff)
Date: 2011-11-20 11:43

Message:
Thabk ypu for yur patch which seems correct


----------------------------------------------------------------------

Comment By: Tito Latini (titola)
Date: 2011-11-15 02:36

Message:
This is a poor solution, the depth of the included files (without bug)
is only 1. For a version with depth > 1, the condition in <<EOF>> rule
has to become

  if (PARM->clearBufferAfterEOF) {
    PARM->lBuffer = 0;
    if (some_pointer_to_include_stack == NULL)
      PARM->clearBufferAfterEOF = 0;
  }

where `some_pointer_to_include_stack' replaces PARM->isInclude.

tito


----------------------------------------------------------------------

Comment By: Tito Latini (titola)
Date: 2011-11-13 09:36

Message:
The previous patch don't works with a $NAME macro
inside a UDO. This example fails

;; foo.txt
opcode test,a,i
  ifn xin
  aout $OSCIL $MAXAMP, 1000, ifn
  xout aout
endop


<CsoundSynthesizer>
<CsInstruments>
#define OSCIL #oscil#
#define MAXAMP #20000#
#include "foo.txt"
instr 1
  aout test 1
  out aout
endin
</CsInstruments>
<CsScore>
f1 0 8192 10 1
i1 0 1
</CsScore>
</CsoundSynthesizer>


The new temporary fix:

diff -ur csound5~/Engine/csound_orc.l csound5/Engine/csound_orc.l
--- csound5~/Engine/csound_orc.l	2011-11-08 13:51:46.000000000 +0100
+++ csound5/Engine/csound_orc.l	2011-11-13 16:53:43.000000000 +0100
@@ -186,8 +186,14 @@
                   return T_INSTR;
                 }
 "endin"         { return T_ENDIN; }
-"opcode"        { return T_UDOSTART; }
-"endop"         { *lvalp = new_token(csound, T_UDOEND); return T_UDOEND; }
+"opcode"        {
+                  PARM->clearBufferAfterEOF = 0;
+                  return T_UDOSTART;
+                }
+"endop"         {
+                  PARM->clearBufferAfterEOF = PARM->isInclude;
+                  *lvalp = new_token(csound, T_UDOEND); return T_UDOEND;
+                }
 
 {LABEL}         { char *pp = yytext;
                   while (*pp==' ' || *pp=='\t') pp++;
@@ -432,6 +438,8 @@
                   yypop_buffer_state(yyscanner);
                   if ( !YY_CURRENT_BUFFER ) yyterminate();
                   printf("End of input; popping to %p\n",
YY_CURRENT_BUFFER);
+                  if (PARM->clearBufferAfterEOF)
+                    PARM->lBuffer = PARM->clearBufferAfterEOF =
PARM->isInclude = 0;
                   n = PARM->alt_stack[--PARM->macro_stack_ptr].n;
                   printf("n=%d\n", n);
                   if (n!=0) {
@@ -530,6 +538,7 @@
     }
     PARM->alt_stack[PARM->macro_stack_ptr].n = 0;
     PARM->alt_stack[PARM->macro_stack_ptr++].s = NULL;
+    PARM->isInclude = PARM->clearBufferAfterEOF = 1;
     yypush_buffer_state(yy_create_buffer( yyin, YY_BUF_SIZE, yyscanner),
yyscanner);
 }
 
diff -ur csound5~/Engine/parse_param.h csound5/Engine/parse_param.h
--- csound5~/Engine/parse_param.h	2011-11-08 13:51:46.000000000 +0100
+++ csound5/Engine/parse_param.h	2011-11-13 16:53:01.000000000 +0100
@@ -30,6 +30,8 @@
     unsigned int macro_stack_ptr;
     char            *xstrbuff;
     int             xstrptr,xstrmax;
+    unsigned char   isInclude;
+    unsigned char   clearBufferAfterEOF;
 } PARSE_PARM;
 
 #define lMaxBuffer (1000)


tito


----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=564599&aid=3437225&group_id=81968

------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure 
contains a definitive record of customers, application performance, 
security threats, fraudulent activity, and more. Splunk takes this 
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
_______________________________________________
Csound-devel mailing list
Csound-devel@lists.sourceforge.net