[Sergiy Radyakin]

Nick, watch for the "ancillary files" route - those files are dropped into current directory, which could be anything (including protected Program Files folder) unless you set it up specifically, before installing the package.

The rest is too cloudy. If you are about to make version 1 of your package, don't worry if it is not going to install smoothly for some users that have batch deployment of Stata in the disconnected mode. You'll get there later. Solve the 95% case first.

Good luck, Sergiy

[Nick Guenther]

I got it working. Check out ensurelib.ado and it's partner ensurelib_aberrance.ado.

Use it like this:

/* load the C extension */
ensurelib_aberrance svm // check for libsvm
program _svm, plugin    // load _svm.plugin, the wrapper for libsvm

program define subroutine_that_needs_libsvm
...
end

ensurelib twiddles PATH if it finds the library in the adopath, and then double-checks that everything works by trying to dlopen() it: in this way, whether you've bundled or not, you know your dependencies are available (with a reasonable error message if not!) before trying to dlopen() the .plugin which needs them.

("aberrance" is just word that MinGW uses for misnamed libraries which are common in Unix-to-Windows ports; that routine handles tacking 'lib' onto libraries on Windows)


Here's a working .pkg file which shows how to distribute everything necessary.

v 3
d svm
d Support Vector Machines for both Classification and Regression
d Distribution-Date: 20150604

F svm/_svm_model2stata.ado
F svm/ensurelib.ado
F svm/ensurelib_aberrance.ado
F svm/export_svmlight.ado
F svm/generate_clone.ado
F svm/import_svmlight.ado
F svm/svm.ado
F svm/svm.sthlp
F svm/svm_export.ado
F svm/svm_import.ado
F svm/svm_predict.ado
F svm/svm_train.ado
F svm/svmlight.sthlp
G LINUX64 svm/bin/LINUX64/_dlopenable.plugin
G LINUX64 svm/bin/LINUX64/_getenv.plugin
G LINUX64 svm/bin/LINUX64/_setenv.plugin
G LINUX64 svm/bin/LINUX64/_svm.plugin
G LINUX64 svm/bin/LINUX64/_svmlight.plugin
G MACINTEL64 svm/bin/MACINTEL64/_dlopenable.plugin
G MACINTEL64 svm/bin/MACINTEL64/_getenv.plugin
G MACINTEL64 svm/bin/MACINTEL64/_setenv.plugin
G MACINTEL64 svm/bin/MACINTEL64/_svm.plugin
G MACINTEL64 svm/bin/MACINTEL64/_svmlight.plugin
G WIN svm/bin/WIN/_dlopenable.plugin
G WIN svm/bin/WIN/_getenv.plugin
G WIN svm/bin/WIN/_setenv.plugin
G WIN svm/bin/WIN/_svm.plugin
G WIN svm/bin/WIN/_svmlight.plugin
G WIN svm/bin/WIN/libsvm.dll
G WIN64A svm/bin/WIN64A/_dlopenable.plugin
G WIN64A svm/bin/WIN64A/_getenv.plugin
G WIN64A svm/bin/WIN64A/_setenv.plugin
G WIN64A svm/bin/WIN64A/_svm.plugin
G WIN64A svm/bin/WIN64A/_svmlight.plugin
G WIN64A svm/bin/WIN64A/libsvm.dll
h _dlopenable.plugin
h _getenv.plugin
h _setenv.plugin
h _svm.plugin
h _svmlight.plugin

Again, the capital letters make Stata install those files to the adopath---even the ones, like the .dlls, that it would normally treat as ancillary because of that hardcoded list of acceptable file extensions. libsvm.dll ends up in C:\ado\plus\l\ on my test system, where findfile locates it and from which ensurelib tacks onto PATH.

Arranging your site to actually have all these files is left as an excercise, but makepluginpkg will turn that set, once you have it arranged, into a .pkg file.

Please let me know if you find this useful, and submit pull requests with suggestions.

__________________________________________________ ______________________________________

ensurelib/_dlopenable.plugin and _get/setenv.plugin should come in separate libraries. If we all start distributing them with our packages bad things will happen to users. Is it definite that Stata's package manager doesn't understand dependencies? What do you normally do in this situation? Just instruct users that they must install another package manually first?

__________________________________________________ ______________________________________

Now, I have another obscure question: does any know if it is possible to access the creturn values from a C plugin? I know you can do this from mata with

mata st_global("c(console)")

but the skimpy C docs offer, at closest

SF_macro_use("c(console)", buf, sizeof(buf)-1)

but I tried that and it just gave me an error code. Is there some sort of name mangling I can do to 'c(console)' to get it?

I don't want to feed its value in from a wrapper because the code I'm interested in checking this in is some print utilities linked into multiple plugins. I won't screw up multiple APIs just for this. Can someone offer me a clue to this undocumented heart of jungle?

[Sergiy Radyakin]

Now, I have another obscure question: does any know if it is possible to access the creturn values from a C plugin?

Do it from a wrapper.
One wrapper should be the gateway for all your utils.
You can branch later as needed.

Best, Sergiy

[Nick Guenther]

Thanks for keeping up with me, Sergiy.

The way I would handle this with a wrapper is, say,

// svm_train.ado

program define svm_train
  creturn_plugin_wrapper _svm_train `0'
end

program define _svm_train
  // [ contents of my current svm_train code ]
end

// creturn_plugin_wrapper.ado
program define creturn_plugin_wrapper
  local c_os_ = "`c(os)'"
  local c_console_ = "`c(console)'"
  // repeat as needed ...
 
  // run the passed program
  `0'
end

// _svm.c
STDLL stata_call(int argc, char argv[]) {
  // ...
  SF_macro_use("_c_console_", buf, sizeof(buf)-1); //the extra underscore comes from the 'local' call
  // ...
}

This seems creaky. Is this what you were thinking? Is there a better way? The linked stutil.c is utilities for internal use, and I am wary breaking encapuslation. I guess this solution isn't as scary as I was picturing last night.

Thanks in advance! You've been very helpful so far.

[Sergiy Radyakin]

Nick, as far as I see the above version of yours should not work. The locals saved by plugin_wrapper will be local to the wrapper and not visible to the program called from it in the last line. That said, I don't know what would that be, since it depends on the calling convention of the original program svm_train.

I think you are overcomplicating things. Note the use of c_local in below:

program define set_plugin_environment
  clocal c_os_   "`c(os)'"
  clocal c_console_   "`c(console)'"
  // repeat as needed ...
end

program define task1
  syntax .....whatever......
  set_plugin_environment
  plugin call MyPluginExternal
end

program define task2
  syntax .....whatever......
  set_plugin_environment
  plugin call MyPluginExternal
end

program define task3
  syntax .....whatever......
  set_plugin_environment
  plugin call MyPluginExternal
end

program define MyPluginExternal, plugin using("MyPlugin.plu")

Best, Sergiy

[Nick Guenther]

Hello again, Sergiy. Yes, I see. I also see that c_local (as it should be spelled, for anyone reading this in the future) is undocumented, but it reaches one frame up the call stack and edits the namespace there. That is sneaky and clever, and could definitely work.

For future reference, a complete working example:

[nguenthe@galleon ~]$ stata

  ___  ____  ____  ____  ____ (R)
 /__    /   ____/   /   ____/
___/   /   /___/   /   /___/   14.0   Copyright 1985-2015 StataCorp LP
  Statistics/Data Analysis            StataCorp
                                      4905 Lakeway Drive
                                      College Station, Texas 77845 USA
                                      800-STATA-PC        http://www.stata.com
                                      979-696-4600        [email protected]
                                      979-696-4601 (fax)

Single-user Stata license expires 11 May 2016:
       Serial number:  xxxxxxxxxxxxx
         Licensed to:  Nick Guenther
                       University of Waterloo

Notes:
      1.  Unicode is supported; see help unicode_advice.

running /home/nguenthe/ado/profile.do ...

. program define set_plugin_environment
  1.   c_local c_os_   "`c(os)'"
  2.   c_local c_console_   "`c(console)'"
  3. end

. macro list
S_FNDATE:       13 Apr 2014 17:45
S_FN:           /usr/local/stata/ado/base/a/auto.dta
S_level:        95
F1:             help advice;
F2:             describe;
F7:             save
F8:             use
S_ADO:          BASE;SITE;.;PERSONAL;PLUS;OLDPLACE
S_CONSOLE:      console
S_FLAVOR:       Intercooled
S_OS:           Unix
S_MACH:         PC (64-bit x86-64)

. set_plugin_environment

. macro list
S_FNDATE:       13 Apr 2014 17:45
S_FN:           /usr/local/stata/ado/base/a/auto.dta
S_level:        95
F1:             help advice;
F2:             describe;
F7:             save
F8:             use
S_ADO:          BASE;SITE;.;PERSONAL;PLUS;OLDPLACE
S_CONSOLE:      console
S_FLAVOR:       Intercooled
S_OS:           Unix
S_MACH:         PC (64-bit x86-64)
_c_console_:    console
_c_os_:         Unix

.

To anyone in the future: we are pretty close to being ready to release this project, and when we do this or something much like it will be integrated and fully working.

[Nick Guenther]

Also, Sergiy, you're right that settings locals like I did shouldn't work, but plugins are special.

See here, what you expect is true: B cannot access A:

. program A        
  1. local x = 1
  2. B
  3. di "A: x = `x'"
  4. end

. program B
  1. di "B: x = `x'"
  2. local x = 2
  3. di "B: x = `x'"
  4. end

. A
B: x =
B: x = 2
A: x = 1

But D.plugin can access C's scope:

. program define C
  1. local c = "zvedz"
  2. program D, plugin
  3. plugin call D, _c
  4. end

. C
locals: read _c = 'zvedz'

. * the _c is because 'locals' are just globals with underscores attached, as per the docs: http://www.stata.com/plugins/#sect8c
. * without it, things don't work:
. program define C2
  1. local c = "manty"
  2. program D, plugin
  3. plugin call D, c
  4. end

. C2
locals: read c = ''

.

// D.c
// to compile this you also need stutil.c and stplugin.c from the repo: https://github.com/kousu/statasvm/
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include "stplugin.h"
#include "stutil.h"

STDLL stata_call(int argc, char *argv[])
{
    
    ST_retcode err = 0;
    char buf[200];
    
    // argument checking
    if(argc != 1) {
        sterror("locals: incorrect number of arguments (%d), can only take 1\n", argc);
        return 1;
    }
  
    err = SF_macro_use(argv[0], buf, sizeof(buf));
    if(err) {
      sterror("locals: unable to read '%s'\n", argv[0]);
      return err;
    }
    stdisplay("locals: read %s = '%s'\n", argv[0], buf);
    return 0;
}

int stata_init() { return 0; }

StataCorp must have realized their C interface was pretty thin---you can't use r(), e() or s() from it---so they allow you to use macros and scalars to get data in and out.

[Sergiy Radyakin]

As I wrote above it was not clear from you example what do you call a wrapper and HOW you call it.
Reading both locals and globals from plugin is documented, intended, and supported by Stata. From the plugins manual:

By macros, we mean both global macros and local macros (local to the program calling the plugin). Internally, global macros and local macros share the same namespace, with the names of local macros preceded by an underscore (_).

What you need to understand is that a plugin is a way to create equivalents of built-in commands. Built-in commands work in the context of the caller, see all of its namespace. In contrast ado files don't. The locals spelled as parameters are really evaluated in the context of the caller before the call is made, and later inaccessible from the callee (the values are of course accessible, through syntax or similar mechanics).

Hence, for a setup A-->B-->P, P will see all globals and all locals of B, it should not see locals of A.
A and B are programs, P is plugin.
In contrast for programs, A-->B-->C, the C will not have access to locals of B.

In reality the fact that built-in commands operate differently then ado-files is not entirely obvious. You can doubt that and say that the built-in still get the chewed content from the caller. For example we write display `"`foo'"' not display foo
However there are built-in commands that work somewhat like that, for example gettoken:

. which gettoken
built-in command:  gettoken
. local foo1 "foo bar"
. gettoken first: foo1
. display `"`first'"'
foo

If you manage to make P see locals of A let me know The idea of course is feasible, but the implementation without binding to a particular Stata executable or overriding other Stata's built-ins would be difficult, at least in older versions of Stata. Luckily you don't need it for your project.

Best, Sergiy

[Nick Guenther]

I'm doing a bit of maintenance on our package svmachines (http://fmwww.bc.edu/repec/bocode/s/svmachines.pkg). I've learned that Stata has simplified the list of supported platforms: compare

Stata 14's https://www.stata.com/manuals14/rnet...kagedirectives, which says

The platform names are WIN64A (64-bit x86-64) and WIN (32-bit x86) for Windows; MACINTEL64 (64-bit Intel, GUI), OSX.X8664 (64-bit Intel, console), MACINTEL (32-bit Intel, GUI), and OSX.X86 (32-bit Intel, console) for Mac; and LINUX64 (64-bit x86-64) and LINUX (32-bit x86) for Unix.

To Stata 15's, https://www.stata.com/manuals15/rnet...kagedirectives

The platform names are WIN64 (64-bit x86-64) and WIN32 (32-bit x86) for Windows; MACINTEL64 (64-bit Intel, GUI) and OSX.X8664 (64-bit Intel, console) for Mac; and LINUX64 (64-bit x86-64) and LINUX (32-bit x86) for Unix.

And https://www.stata.com/manuals/rnet.p...kagedirectives, which presumably is current for Stata 16

The platform names are WIN64 (64-bit x86-64) for Windows; MACINTEL64 (64-bit Intel, GUI) and OSX.X8664 (64-bit Intel, console) for Mac; and LINUX64 (64-bit x86-64) for Unix.

Originally, I supported 5 of those platforms in my package:

Windows 64-bit x86-64 = WIN64A
Windows 32-bit x86 = WIN
macOS 64-bit Intel, GUI = MACINTEL64
Linux 64-bit x86-64 = LINUX64
Linux 32-bit x86 = LINUX

But now with Stata 15 and 16, some of these have been renamed, so now covering all versions and all platforms looks like:

Windows 64-bit x86-64 = WIN64A, WIN64
Windows 32-bit x86 = WIN, WIN32
macOS 64-bit Intel, GUI = MACINTEL64
Linux 64-bit x86-64 = LINUX64
Linux 32-bit x86 = LINUX

Which translates to **5** platform-specific builds but **7 lines** *per* plugin in the .pkg file:

G LINUX      pkg_linux32.plugin     pkg.plugin
G LINUX64  pkg_linux64.plugin     pkg.plugin
G MACINTEL64   pkg_mac64.plugin       pkg.plugin
G WIN        pkg_win32.plugin       pkg.plugin
G WIN32    pkg_win32.plugin       plg.plugin
G WIN64    pkg_win64.plugin       pkg.plugin
G WIN64A  pkg_win64.plugin       pkg.plugin

Thanks to everyone in this thread for your help over these years.