Importing C code in Stata and Nonlinear optimization

[Michele Santacatterina]

Dear Statalist,

I am trying to import some C code from the NLopt library (http://ab-initio.mit.edu/wiki/index.php/NLopt) into Stata.
I followed this tutorial (http://www.stata.com/plugins/) to create the dll library containing the files stplugin.h, stplugin.c, and myfile.c.

I managed to run without errors:

program nlopt, plugin using("C:\ado\personal\nlopt.dll")


Nonetheless, i got the error r(9998) once i try to call the plugin:

. plugin call nlopt1
r(9998);

end of do-file


Any idea how to make this work?
I tried to run the example hello.c and it works smoothly.

Thank you very much in advance,

MS


p.s if anyone knows a Stata command for nonlinear optimization (specifically sequential quadratic programming) please let me know! :D

[Michele Santacatterina]

I would like to add that i built the dll library using Visual studio 2015.

[Nick Guenther]

It is good to meet another person working on C plugins!

Since your crash is happening after the loading of the plugin, it must be happening sometime during stata_call(). I can't tell without code what the problem might be. You should pepper stata_call() with SF_display() calls.

By the way, "with Visual Studio" isn't really enough information. It is helpful, but what switches did you use? Do you have optimizations on? What did you link against? What version of C did you compile with?



Another thing to be aware of, which is not at all explained at /plugins, is that to make wrappers, at least to make ones that will work on other peoples' machines, you must statically link against the wrapped code. You can check if you've made a dynamic link instead with MSVC with

dumpbin /dependents

(under MinGW you can use

objdump -x | grep DLL

and under regular POSIX you can use

readelf -d

). However, since you say the plugin load worked this cannot (should not?) be your current problem---whenever I've had DLL mistakes my plugins have refused to load, because DLLs resolve all dependencies at LoadLibrary(). Oh, unless you've compiled with Delayed Loading, which would entirely explain the problem; if you copy the DLL to your *current working directory*, wherever Stata is cd'd to at the time you run "program, plugin", and try again, and it works, then this is your problem.

I have worked out a hack to allow me to distribute a DLL alongside my code. If you would like to make use of it as your project progresses go for it: it is BSD licensed. It is a bit messy though, being inline in my project at the moment (but, I think, all its components should be independent of my project). It would be great to collaborate on factoring this into a properly separate library in its own right.

[Michele Santacatterina]

Dear Nick,

Thank you for your reply!

It is good to meet another person working on C plugins!

Since your crash is happening after the loading of the plugin, it must be happening sometime during stata_call(). I can't tell without code what the problem might be. You should pepper stata_call() with SF_display() calls.

By the way, "with Visual Studio" isn't really enough information. It is helpful, but what switches did you use? Do you have optimizations on? What did you link against? What version of C did you compile with?

I checked some default properties of visual studio and the only thing i can retrieve is that optimization is disabled.

Another thing to be aware of, which is not at all explained at /plugins, is that to make wrappers, at least to make ones that will work on other peoples' machines, you must statically link against the wrapped code. You can check if you've made a dynamic link instead with MSVC with

dumpbin /dependents

(under MinGW you can use

objdump -x | grep DLL

and under regular POSIX you can use

readelf -d

). However, since you say the plugin load worked this cannot (should not?) be your current problem---whenever I've had DLL mistakes my plugins have refused to load, because DLLs resolve all dependencies at LoadLibrary(). Oh, unless you've compiled with Delayed Loading, which would entirely explain the problem; if you copy the DLL to your *current working directory*, wherever Stata is cd'd to at the time you run "program, plugin", and try again, and it works, then this is your problem.

I have worked out a hack to allow me to distribute a DLL alongside my code. If you would like to make use of it as your project progresses go for it: it is BSD licensed. It is a bit messy though, being inline in my project at the moment (but, I think, all its components should be independent of my project). It would be great to collaborate on factoring this into a properly separate library in its own right.

Nonetheless, as you said the I can load the plugin, then the problem should be somewhere else. I have the impression that my issue is due to the fact that the C code I am trying to import in Stata (as a dll) does not have the proper Stata structures, such as those written in stplugin.h. I tried to make a simple C code without those Stata structures, building the dll and loading it into Stata and I have the same issue.

Then, what I will do for my specific problem is:

1. install (properly) NLopt in Windows;
2. try to run (from Stata) this example using the Stata structures;
3. try to generalize it to a more general problem.

...and (i guess) for point 3 your hack Nick might be useful (am i wrong?).

For now the main issue remains to wrap the C code of NLoptr into Stata. If you have any advice on how to do this It would be really appreciated!

Thank you again!

MS

[Michele Santacatterina]

Ok I managed to make the example in http://ab-initio.mit.edu/wiki/index....in_C.2FC.2B.2B working.
Following all the steps I did to make this work.

1. Download Visual Studio and install it (https://www.visualstudio.com/).
2. Download the pre-built DLL library from the NLopt website (http://ab-initio.mit.edu/wiki/index....opt_on_Windows).
3. From the readme-windows file (inside nlopt-2.4.2-dll64.zip) it says: “In order to link to this .dll files from Visual C++, you need to create a .lib "import libraries" for it, and can do so with the "lib" command that comes with VC++. In particular, run: lib /def:libnlopt-0.def”
   1. Run vsvars32.bat:
      1. From the Start menu, open the Developer Command Prompt for VS2012.
      2. Change to the Program Files\Microsoft Visual Studio Version\Common7\Tools or Program Files (x86)\Microsoft Visual Studio Version\Common7\Tools subdirectory of your installation.
      3. Run VSVARS32.bat by typing VSVARS32.
   2. Execute command "lib /def:libnlopt-0.def"
   3. As result you will get a libnlopt-0.lib and a libnlopt-0.exp file.
4. Download http://www.stata.com/plugins/stplugin.h and http://www.stata.com/plugins/stplugin.c.
5. Download the example Source.c from https://www.dropbox.com/s/s99dfs98qy1rh2x/Source.c?dl=0. What this code does is the example in http://ab-initio.mit.edu/wiki/index....in_C.2FC.2B.2B adapted with the Stata structures (e.g. double à ST_double).
6. Create a new folder and add into it: libnlopt-0.lib, libnlopt-0.exp, libnlopt-0.dll, nlopt.h (from the zip package) stplugin.h, stplugin.c and Source.c.
7. Open Visual Studio:
   1. Select File > New > Project.
   2. Expand Visual C++, select Win32 Project from Templates.
   3. Under the Win32 Application Wizard, select Application Settings and click on DLL. You should select Empty project to open a project with no existing files.
   4. Under the Solution Explorer, right-click on Source files, and select Add > Add Existing Item.
   5. Add the files libnlopt-0.lib, libnlopt-0.exp, libnlopt-0.dll, nlopt.h, stplugin.h, stplugin.c and Source.c
   6. Build the DLL by selecting Build. If all goes well (it should), you will now find the file name_project.dll within the Debug subdirectory of your project directory.
8. Now, take name_project.dll and copy it into c:\ado\personal\.
9. Open Stata and in a new do/ado file type:

program nlopt, plugin using("C:\ado\personal\name_project.dll")
plugin call nlopt

It should give you a result like: found minimum at f(0.333333,0.296296) = 0.5443310474
that is the optimal solution for this nonlinear optimization problem.


The next step will be to give in input some info like the type of algorithm, initial points, etc. It would be nice to give as input all the constraints of the NLo problem (from Stata), but for now i will write them down in C in my Source.c file (I am interested in one specific problem right now).

Well, hope this will help somebody! :D

I will keep you update!

MS

[Nick Guenther]

It's great that you got it running. I am curious: when you run

program nlopt, plugin using("C:\ado\personal\name_project.dll")

where is

libnlopt-0.dll

?

Under Windows, the current working directory is always on the library search path (which incidentally is a huge source of security bugs), which is what I expect is happening here. Be careful of this, because it *will* trip you up (which is what my hack solves) Is it still loadable if you try to load it from any other path? Try

cd c:\
program nlopt, plugin using("C:\ado\personal\name_project.dll")

and tell us what happens.

*Another* thing which will trip you up is architecture: you've compiled at 32-bit nlopt and a 32-bit plugin, so can I assume you're running 32-bit Stata? Perhaps your problem before was that had 64-bit libnlopt. It is super difficult to debug these sorts of things because they break at low levels and don't have a lot of avenues for reporting errors: it's hard enough on Unix, and I assume Windows is even trickier.

Once you are able to do the (dynamic) linking against libnlopt, actually doing the wrapper is the fun part which requires you to imagine what a good Stata-esque API looks like for this task.

Good luck!

[Michele Santacatterina]

It's great that you got it running. I am curious: when you run

program nlopt, plugin using("C:\ado\personal\name_project.dll")

where is

libnlopt-0.dll

?

Under Windows, the current working directory is always on the library search path (which incidentally is a huge source of security bugs), which is what I expect is happening here.

You got it right! Sorry Nick, I forgot to say that I added

libnlopt-0.dll

into C:\Windows\System32\.

Be careful of this, because it *will* trip you up (which is what my hack solves)

What do you think might happen?
(just to let you know, for now I need it working on my PC only. In the future I would like to make a Stata command that everybody could use).

Is it still loadable if you try to load it from any other path? Try

cd c:\
program nlopt, plugin using("C:\ado\personal\name_project.dll")

and tell us what happens.

Apparently it works.

*Another* thing which will trip you up is architecture: you've compiled at 32-bit nlopt and a 32-bit plugin, so can I assume you're running 32-bit Stata? Perhaps your problem before was that had 64-bit libnlopt. It is super difficult to debug these sorts of things because they break at low levels and don't have a lot of avenues for reporting errors: it's hard enough on Unix, and I assume Windows is even trickier.

Yes, actually Stata, the plugin and nlopt are at 64-bit, but yes I got several issues because of the architecture.

Once you are able to do the (dynamic) linking against libnlopt, actually doing the wrapper is the fun part which requires you to imagine what a good Stata-esque API looks like for this task.

Good luck!

Thank you again! :D