BUG? Unable to nest multi-line mata command within false if statement

[Mauricio Caceres]

I have a program where a multi-line mata command appears inside an if condition. When the condition evaluates to false I get an error; it seems the closing brace of the multi-line mata statement is misinterpreted by Stata as the closing brace of the if condition. This does not appear to happen with other commands (e.g. nested if conditions, nested multi-line quietly, etc.) Here's a MWE tested in Stata 16/MP and Stata17/MP

program test
    args condition
    if ( `condition' ) {
        mata {
            "foo"
            "bar"
        }
    }
end
test 1 // no error
test 0 // error

Output is

. test 1 // no error
 foo
 bar

. test 0 // error
} is not a valid command name
r(199);

LMK if I am doing something wrong or if this is a mistake with the Stata parser.

[William Lisowski]

Consider the following demonstration.

program test
    args condition
    display "condition `condition'"
    if ( `condition' ) {
        display "within conditional before mata block"
        mata {
            "foo"
            "bar"
        }
        display "within conditional after mata block"
    }
end
test 1 // no error
test 0 // error

. test 1 // no error
condition 1
within conditional before mata block
  foo
  bar
within conditional after mata block

. test 0 // error
condition 0
within conditional after mata block
} is not a valid command name
r(199);

You can see that Stata interpreted the end of the mata { } block as the end of the if { } block.

That is because when if skips the conditional code, Stata makes no attempt to parse what follows, just looking for a closing right brace, not understanding that the mata { line introduced another left brace. So it believes the conditional ends before it actually does.

This is true in general. For example, If you want

• a block of Mata code terminated by end

to run within

• a Stata block surrounded by braces {} - typically a program, but in fact within any brace-enclosed block of code

you need to

• define your Mata code as a Mata function outside the brace-enclosed block
• call the Mata function within the brace-enclosed block

Even though your "end" is meant to terminate a Mata block, at the time Stata is parsing the loop, the fact that the "end" belongs to Mata and not to Stata is not recognized, and causes Stata to react incorrectly.

So it's not a bug, it's a feature.

[Mauricio Caceres]

That is because when if skips the conditional code, Stata makes no attempt to parse what follows, just looking for a closing right brace,

William Lisowski This is not true, actually, as I noted in my original post. To be specific, take the following examples:

program test_quietly
    if ( 0 ) {
        quietly {
            "foo"
            "bar"
        }
    }
end

program test_capture
    if ( 0 ) {
        capture {
            "foo"
            "bar"
        }
    }
end

program test_noisily
    if ( 0 ) {
        noisily {
            "foo"
            "bar"
        }
    }
end

program test_doubleif
    if ( 0 ) {
        if ( 1 ) {
            "foo"
            "bar"
        }
    }
end

program test_while
    if ( 0 ) {
        while ( 0 ) {
            "foo"
            "bar"
        }
    }
end

program test_class
    if ( 0 ) {
        class foo {
            double foo
            double bar
        }
    }
end

test_quietly
test_capture
test_noisily
test_doubleif
test_while
test_class

If, as you say, Stata made no attempt to parse what followed and simply looked for a closing brace, all of these would give errors, but they don't. Only mata (and python) behave this way as best I could tell. (Incidentally, it's a bit weird that in my example, if you run "test_class" twice you actually get an error, as Stata complains that class foo is already defined, but it shouldn't be, further evidencing Stata is doing some parsing within false if blocks.)

[Mauricio Caceres]

PS: The same thing happens if I pass a multi-line mata command inline, but not any other command.

program testinline_mata
    args condition
    if ( `condition' ) mata {
        "foo"
    }
end

program testinline_qui
    args condition
    if ( `condition' ) qui {
        disp "foo"
    }
end

program testinline_class
    args condition
    if ( `condition' ) class bar {
        double bar
    }
end

testinline_qui 0 // no error
testinline_qui 1 // no error

testinline_class 0 // no error
testinline_class 1 // error; somehow class bar was already defined

testinline_mata 1 // no error
testinline_mata 0 // error, as "mata {" gets ignored Stata thinks "if" is done, so "foo" is run outside of "if"

[William Lisowski]

This is not true, actually, as I noted in my original post.

That is true, actually, because I was explaining the behavior of the code in post #2 and in post #1, containing an embedded Mata block. I was not making the overarching pronouncement you inferred.

[Leonardo Guizzetti]

To illustrate William's point in #2, you can examine the output of -set trace on- for the failed condition.

. test 0 // error
  ---------------------------------------------------------------------------------------------------------------------------------------------------- begin test ---
  - args condition
  - display "condition `condition'"
  = display "condition 0"
condition 0
  - if ( `condition' ) {
  = if ( 0 ) {
    display "within conditional before mata block"
    mata {
    "foo"
    "bar"
    }
  - display "within conditional after mata block"
within conditional after mata block
  - }

We can see that the red line is executed, despite the condition not being true. Therefore, Stata did just look ahead to the first closing brace and jump back into execution from that point. You can also circumvent the error noted in #1 by prefixing each mata command with the mata prefix, instead of trying to use a mata block.

Note the examples given in #4 are abuses of the -if- syntax and should not in general be trusted to work. Multiple commands must be enclosed in braces as indicated in the output of -help ifcmd-.

[Mauricio Caceres]

That is true, actually, because I was explaining the behavior of the code in post #2 and in post #1, containing an embedded Mata block. I was not making the overarching pronouncement you inferred.

Ah, I now see what you meant. I was mainly confused because the fact "if" only works this way with mata and python is what drove me to make this post, so I wanted to clarify.

[Mauricio Caceres]

You can also circumvent the error noted in #1 by prefixing each mata command with the mata prefix, instead of trying to use a mata block.

Yes, you can also embed the if condition in the mata block and it will also work. You can also write a separate mata function that works as a wrapper for the block, which was suggested in #2, and that will work as well. I was mainly trying to figure out if this was a bug or expected behavior for some reason. William Lisowski, I now understand, argues the inconsistent behavior of "if" with respect to mata/python vs other commands with bracketed input is a feature, but I don't follow the argument.

Note the examples given in #4 are abuses of the -if- syntax and should not in general be trusted to work. Multiple commands must be enclosed in braces as indicated in the output of -help ifcmd-.

Indeed, I normally do not use this syntax. I was further trying to illustrate the point that "if" seems to selectively parse (and, as the class example shows, possibly partially execute) false blocks.

[William Lisowski]

It is a feature in that it is part of the design of Stata. It may not be a design you appreciate, but it is functioning as designed.

Neither Mata nor Python are Stata, and when the mata { command is found within a block of Stata code being bypassed, Stata makes no attempt to invoke Mata in order to have it scan the following code without actually running it and return control to Stata once it hits the correct right brace that brings the block to an end - if such a thing were even possible. Same is true for Python, and is somewhat more obvious, since it's more obviously not part of Stata.

In a discussion on Statalist some time ago, which I cannot quickly locate, a StataCorp employee explained that a block of code enclosed within braces was in many respects the same as a Stata program. From that I take it that this is expected behavior, not a BUG or even a bug.

So in that sense, your if { command is like defining a Stata program, and nothing good ever comes of trying to define a Mata function within a Stata program definition.

With that, I've said all I have to say on this.

[Mauricio Caceres]

It is a feature in that it is part of the design of Stata. It may not be a design you appreciate, but it is functioning as designed.

Neither Mata nor Python are Stata, and when the mata { command is found within a block of Stata code being bypassed, Stata makes no attempt to invoke Mata in order to have it scan the following code without actually running it and return control to Stata once it hits the correct right brace that brings the block to an end - if such a thing were even possible. Same is true for Python, and is somewhat more obvious, since it's more obviously not part of Stata.

In a discussion on Statalist some time ago, which I cannot quickly locate, a StataCorp employee explained that a block of code enclosed within braces was in many respects the same as a Stata program. From that I take it that this is expected behavior, not a BUG or even a bug.

So in that sense, your if { command is like defining a Stata program, and nothing good ever comes of trying to define a Mata function within a Stata program definition.

With that, I've said all I have to say on this.

I see what you mean. I may not like it but I can follow this argument. While python is certainly external, I think of mata as just another part of Stata. But you are right that internally mata is being treated differently.

I disagree it's not a bug, because the behavior of "if" should be consistent, but it seems like it might not be fixable and I'll agree there's a coherent reason for that.

[Clyde Schechter]

The point that any group of commands surrounded by curly braces are, in Stata, a program, is under-appreciated. But Stata is consistent about this. For example, if you redefine a local macro in a block of commands, its new definition holds only between those braces, and the original definition is restored outside them.

. clear*

.
. forvalues i = 1/5 {
  2.     display _newline "Outer loop i = `i'"
  3.     local j = `i'
  4.     display "Outer loop j = `j'"
  5.     forvalues i = 1/3 {
  6.         display "Inner loop i = `i'"
  7.         local j = `i'
  8.         display "Inner loop j = `j'"
  9.     }
 10. }

Outer loop i = 1
Outer loop j = 1
Inner loop i = 1
Inner loop j = 1
Inner loop i = 2
Inner loop j = 2
Inner loop i = 3
Inner loop j = 3

Outer loop i = 2
Outer loop j = 2
Inner loop i = 1
Inner loop j = 1
Inner loop i = 2
Inner loop j = 2
Inner loop i = 3
Inner loop j = 3

Outer loop i = 3
Outer loop j = 3
Inner loop i = 1
Inner loop j = 1
Inner loop i = 2
Inner loop j = 2
Inner loop i = 3
Inner loop j = 3

Outer loop i = 4
Outer loop j = 4
Inner loop i = 1
Inner loop j = 1
Inner loop i = 2
Inner loop j = 2
Inner loop i = 3
Inner loop j = 3

Outer loop i = 5
Outer loop j = 5
Inner loop i = 1
Inner loop j = 1
Inner loop i = 2
Inner loop j = 2
Inner loop i = 3
Inner loop j = 3

.
end of do-file

Note: It would be terrible programming practice to do this in a real-world situation. I'm just demonstrating that the rule that the scope of a local macro defined in a program is restricted to that program, and not to any subsequently called programs, nor back into the calling program is followed consistently with the convention that code embedded between curly braces is a program.