Problem with entropyetc

[Paris Rira]

Dear Profs and colleagues,

I am going to generate HHI, Shannon, etc by using entropyetc, however, it says nothing to do.
So I explain what I seek, perhaps I am wrong in what I am doing.
I would like to generate diversity indices for a categorical variable Edugroup for only foreigners, data has local and foreigners. When foreign==1 is foreigners. (I don't wanna use keep if foreign==1)
NPC_FIC is firm ID. year 2010-2019.

* Example generated by -dataex-. For more info, type help dataex
clear
input double(year NPC_FIC) float(foreign Edugroup)
2010 500000001 1 2
2010 500000001 0 2
2010 500000001 1 2
2010 500000002 0 2
2010 500000002 12
2010 500000002 0 2
2010 500000002 0 2
2011 500000002 0 2
2011 500000002 1 4
2011 500000002 0 2
2011 500000002 0 2
2012 500000002 0 2
2012 500000002 0 2
2012 500000002 1 4
2012 500000002 0 2
2012 500000002 1 2
2013 500000002 0 2
2013 500000002 1 2
2013 500000002 0 2
2013 500000002 0 2
2014 500000002 0 2
2015 500000002 0 2
2015 500000002 0 2
2016 500000002 0 2
2016 500000002 0 2
2017 500000002 0 2
2017 500000002 0 2
2010 500000033 0 2
2010 500000033 1 2
2011 500000033 0 2
2011 500000033 0 2
2012 500000033 1 2
2012 500000033 0 2
2013 500000033 0 2
2014 500000033 0 2
2015 500000033 0 2
2015 500000033 0 4
2016 500000033 0 2
2017 500000033 0 2
2018 500000033 0 2
2019 500000033 0 2
2010 500000050 0 1
2010 500000050 1 2
2011 500000050 0 2
2012 500000050 1 2
2013 500000050 1 2
2014 500000050 0 2
2014 500000050 0 2
2015 500000050 0 2
2015 500000050 0 2
2019 500000073 0 2
2010 500000083 0 3
2011 500000083 0 3
2012 500000083 0 3
2013 500000083 0 3
2014 500000083 0 3
2015 500000083 0 3
2016 500000083 0 3
2017 500000083 0 3
2018 500000083 0 3
2018 500000083 0 3
2019 500000083 0 3
2019 500000083 0 3
2015 500000101 0 2
2016 500000101 0 2
2017 500000101 0 2
2018 500000101 0 2
2019 500000101 0 2
2010 500000104 1 2
2011 500000106 0 2
2011 500000113 0 3
2012 500000113 0 2
2013 500000113 0 2
2014 500000113 0 2
2010 500000119 0 2
2010 500000119 1 1
2010 500000119 1 2
2010 500000119 1 2
2010 500000119 0 2
2010 500000119 0 2
2010 500000119 0 2
2010 500000119 0 2
2011 500000119 0 2
2011 500000119 0 2
2011 500000119 0 2
2011 500000119 0 2
2011 500000119 0 1
2011 500000119 0 2
2011 500000119 0 2
2011 500000119 0 2
2011 500000119 0 2
2013 500000119 0 1
2013 500000119 0 2
2012 500000119 0 3
2012 500000119 0 2
2013 500000119 0 2
2012 500000119 0 2
2012 500000119 1 2
2012 500000119 0 2
2015 500000121 0 4
end

. tab Edugroup

   Edugroup |      Freq.     Percent        Cum.
------------+-----------------------------------
          1 |  2,906,475       10.91       10.91
          2 | 11,098,633       41.66       52.57
          3 |  7,393,341       27.75       80.32
          4 |  5,243,652       19.68      100.00
------------+-----------------------------------
      Total | 26,642,101      100.00

. tab foreign

    foreign |      Freq.     Percent        Cum.
------------+-----------------------------------
          0 | 25,295,665       94.95       94.95
          1 |  1,346,436        5.05      100.00
------------+-----------------------------------
      Total | 26,642,101      100.00

If does not work with hhi.

g id= group(NPC_FIC)

. hhi Edugroup, by(year id) if foreign==1
option if not allowed
r(198);


. entropyetc Edugroup, by(id year foreign)
nothing to do

Any ideas appreciated.
Cheers,
Paris

[Nick Cox]

entropyetc and hhi are from SSC, as you are asked to explain (FAQ Advice #12). After hundreds of posts here, you should be familiar with that request!

entropyetc allows generation of variables and listing of results, but you must specify at least one of those; if not, then you have given entropyetc nothing to do and it bails out. Usually you don't want both. See the help for examples of its use. You have downloaded the current version of entropyetc, but may be reading posts with examples of its use with an earlier version. The syntax has changed, as explained in #11 of https://www.statalist.org/forums/for...lable-from-ssc

The second problem is a violation of standard Stata syntax and nothing to with hhi. An if qualifier cannot appear among the options.

You have another problem you haven't mentioned which is much more subtle and insidious. There is an undocumented group() function that works with generate, but what you presumably wanted to do was to use egen instead. However, NPC_FIC is already the variable you need, and the command is not needed any way. See
https://www.statalist.org/forums/for...group-function and https://www.stata.com/statalist/arch.../msg00406.html The 2007 post may be the fullest version of the story.

There is a typo in your data example and the corresponding observation is ignored in the final calculation. You should check your real data for that and similar errors.

This code works and should be closer to what you want. I have sprinkled results among the code.

* Example generated by -dataex-. For more info, type help dataex
clear
input double(year NPC_FIC) float(foreign Edugroup)
2010 500000001 1 2
2010 500000001 0 2
2010 500000001 1 2
2010 500000002 0 2
2010 500000002 12
2010 500000002 0 2
2010 500000002 0 2
2011 500000002 0 2
2011 500000002 1 4
2011 500000002 0 2
2011 500000002 0 2
2012 500000002 0 2
2012 500000002 0 2
2012 500000002 1 4
2012 500000002 0 2
2012 500000002 1 2
2013 500000002 0 2
2013 500000002 1 2
2013 500000002 0 2
2013 500000002 0 2
2014 500000002 0 2
2015 500000002 0 2
2015 500000002 0 2
2016 500000002 0 2
2016 500000002 0 2
2017 500000002 0 2
2017 500000002 0 2
2010 500000033 0 2
2010 500000033 1 2
2011 500000033 0 2
2011 500000033 0 2
2012 500000033 1 2
2012 500000033 0 2
2013 500000033 0 2
2014 500000033 0 2
2015 500000033 0 2
2015 500000033 0 4
2016 500000033 0 2
2017 500000033 0 2
2018 500000033 0 2
2019 500000033 0 2
2010 500000050 0 1
2010 500000050 1 2
2011 500000050 0 2
2012 500000050 1 2
2013 500000050 1 2
2014 500000050 0 2
2014 500000050 0 2
2015 500000050 0 2
2015 500000050 0 2
2019 500000073 0 2
2010 500000083 0 3
2011 500000083 0 3
2012 500000083 0 3
2013 500000083 0 3
2014 500000083 0 3
2015 500000083 0 3
2016 500000083 0 3
2017 500000083 0 3
2018 500000083 0 3
2018 500000083 0 3
2019 500000083 0 3
2019 500000083 0 3
2015 500000101 0 2
2016 500000101 0 2
2017 500000101 0 2
2018 500000101 0 2
2019 500000101 0 2
2010 500000104 1 2
2011 500000106 0 2
2011 500000113 0 3
2012 500000113 0 2
2013 500000113 0 2
2014 500000113 0 2
2010 500000119 0 2
2010 500000119 1 1
2010 500000119 1 2
2010 500000119 1 2
2010 500000119 0 2
2010 500000119 0 2
2010 500000119 0 2
2010 500000119 0 2
2011 500000119 0 2
2011 500000119 0 2
2011 500000119 0 2
2011 500000119 0 2
2011 500000119 0 1
2011 500000119 0 2
2011 500000119 0 2
2011 500000119 0 2
2011 500000119 0 2
2013 500000119 0 1
2013 500000119 0 2
2012 500000119 0 3
2012 500000119 0 2
2013 500000119 0 2
2012 500000119 0 2
2012 500000119 1 2
2012 500000119 0 2
2015 500000121 0 4
end

entropyetc NPC_FIC if foreign==1, by(year) gen(4=wanted)

egen tag = tag(year) if foreign == 1

list year wanted if tag, noobs

  +---------------+
  | year   wanted |
  |---------------|
  | 2010      .25 |
  | 2011        1 |
  | 2012      .28 |
  | 2013       .5 |
  +---------------+




. tab year NPC_FIC if foreign == 1

           |                              NPC_FIC
      year |  5.00e+08   5.00e+08   5.00e+08   5.00e+08   5.00e+08   5.00e+08 |     Total
-----------+------------------------------------------------------------------+----------
      2010 |         2          0          1          1          1          3 |         8
      2011 |         0          1          0          0          0          0 |         1
      2012 |         0          2          1          1          0          1 |         5
      2013 |         0          1          0          1          0          0 |         2
-----------+------------------------------------------------------------------+----------
     Total |         2          4          2          3          1          4 |        16

[Paris Rira]

Prof Nick,
Thank you so much for getting back to me.

I already read the threads indeed though I don't understand it, still, I don't know what should I exactly write in gen(4=wanted). I would like to generate several indices for several categorial variables i.g. Edugroup, nationality based on year and firm ID(which is not listed in this thread, etc)


In https://www.statalist.org/forums/for...th-2-variables #3 or inhttps://www.statalist.org/forums/for...lable-from-ssc you entropyetc rep78 without gen option and the result is a table of several indices.

what I need is that table with several indices.( for each variable separately several indices).

[Nick Cox]

I've already explained in #2 -- and pointed to a post which explains in more detail -- that the syntax of entropyetc has changed, so previous posts are no longer a guide to what works with the latest version of the command.

Otherwise, I don't understand how you're asking a different question for which an answer is not evident after reading the help carefully and trying some experiments.

Wanting calculations for different scales or categorizations is fine by me, but entropyetc does not offer ways of working with two or more different kinds of breakdown at once.

[Paris Rira]

Now I quite understand but not fully. There are several questions:
1- entropyetc has already subtracted from 1 or shall I do it by myself?
2- printing tables for large obs as mine (more than 1 million) is not useful. When I apply qu only one variable appears in my dataset, while in the table there are Shannon H, exp(H), Simpson, 1/Simpson which I am interested in generating all of them in my dataset as well. So this one which is generated what index is? Shannon? Simpson? or ? I would like to have all of them. What is the command for it in case exists?
3- The result for Simpson from

 entropyetc Educ, by( year NPC_FIC ) generate(4=H)

should be equal with

 hhi Educ, by(year NPC_FIC )

right? But I couldn't reach to the same results.

I really appreciate your assistance, Prof Nick.

[Nick Cox]

1. I think you're alluding to the measure calculated by entropyetc which is documented in its help as being the sum of squared probabilities. If you want its complement, you need to calculate that yourself. (If the help isn't clear, the code can always be consulted.)

2. Indeed, for large datasets, or more crucially, those with many categories, listings or tabulations are not helpful, which is precisely why tabulations can be suppressed in entropyetc.

3. hhi wasn't written by me so I don't feel responsible for it. If its results puzzle you, then post a concrete example with a simple dataset. "I don't get the same results" is not a good problem statement.

But entropyetc treats the values of its main argument as defining the distinct levels of a categorical variable. That's quite different from any command that interprets them as giving the frequencies of those categories.

The difference is one reason why different commands exist in this territory.

[Paris Rira]

2. Indeed, for large datasets, or more crucially, those with many categories, listings or tabulations are not helpful, which is precisely why tabulations can be suppressed in entropyetc.

How can I generate all of those indices (Shannon H, exp(H), Simpson, 1/Simpson) in my dataset? Apparently, it should generate one column for each of Shannon H, exp(H), Simpson, 1/Simpson, but when I apply "quietly" only one column is generated in my dataset, and due to large numbers I am not able to track and distinguish which is which.

I think you're alluding to the measure calculated by entropyetc which is documented in its help as being the sum of squared probabilities....

Yes I need to compute proportions. Beacuse I have to compute only for foreigners ( i have local and foreigners) without dropping local from the data, so computing portions is trouble to me. As this code does not consider "if"condition, i don't know why

by NPC_FIC year nationality, sort: gen this_nationality = _N if foreign==1
by NPC_FIC year (nationality): gen everyone = _N  if foreign==1
gen ethnic = this_nationality/everyone

where I need to make diversity index for nationality which is a categorial viable in 8 categories.

In help says that "generate() specifies the creation of between one and five new variables from results." so it should generate in my dataset. while when I do gen(4=H) only one column generates in my dataset. Is not supposed to make 4 columns to me?

[Nick Cox]

All of these questions are answerable by studying the help or through little experiments.

For columns read variables.

My advice: Don't try to learn how a program works with a big dataset. Do learn how a program works with a small(ish) dataset. The datasets used as examples in the help for entropyetc are all much smaller than yours appears to be.

With the option gen(4=H) you will get just one new variable, the one you asked for. If you want more variables, you need to ask for them. There is an example in the help.

quietly should make precisely no difference to which variables are created.

entropyetc supports if. What leads you to think it doesn't? Here is a silly example.

 sysuse auto, clear
(1978 automobile data)

. entropyetc rep78, list

  +-----------------------------------------------------------+
  |       distinct   Shannon H   exp(H)   Simpson   1/Simpson |
  |-----------------------------------------------------------|
  | all          5       1.358    3.888     0.297       3.369 |
  +-----------------------------------------------------------+

. entropyetc rep78 if foreign == 1, list

  +-----------------------------------------------------------+
  |       distinct   Shannon H   exp(H)   Simpson   1/Simpson |
  |-----------------------------------------------------------|
  | all          3       1.004    2.730     0.388       2.579 |
  +-----------------------------------------------------------+

[Paris Rira]

Prof Nick,
Sorry, I was not able to express my question properly.
I don't want to have several variables at once, I meant generating several INDICES as it is shown in the table, I wanna have all of them in my dataset as well. In the above example, the table has
Shannon H (1.004), exp(H)(2.730), Simpson(0.388), 1/Simpson(2.579), right, do they appear all of them in your dataset as well?

[Nick Cox]

Already answered, at least by example. You get what you ask for. If you want a table or list and new variables, ask for both. But a table or list shows all results; generate() only generates what you ask for.

What you're calling indices is up to you. It's not a term I use in this context.

. 
. sysuse auto, clear
(1978 automobile data)

. entropyetc rep78, list

  +-----------------------------------------------------------+
  |       distinct   Shannon H   exp(H)   Simpson   1/Simpson |
  |-----------------------------------------------------------|
  | all          5       1.358    3.888     0.297       3.369 |
  +-----------------------------------------------------------+

. entropyetc rep78 if foreign == 1, list gen(2=H 4=R)

  +-----------------------------------------------------------+
  |       distinct   Shannon H   exp(H)   Simpson   1/Simpson |
  |-----------------------------------------------------------|
  | all          3       1.004    2.730     0.388       2.579 |
  +-----------------------------------------------------------+
(53 missing values generated)
(53 missing values generated)

. l  H R if foreign

     +---------------------+
     |        H          R |
     |---------------------|
 53. | 1.004242   .3877551 |
 54. | 1.004242   .3877551 |
 55. | 1.004242   .3877551 |
 56. | 1.004242   .3877551 |
 57. | 1.004242   .3877551 |
     |---------------------|
 58. | 1.004242   .3877551 |
 59. | 1.004242   .3877551 |
 60. | 1.004242   .3877551 |
 61. | 1.004242   .3877551 |
 62. | 1.004242   .3877551 |
     |---------------------|
 63. | 1.004242   .3877551 |
 64. |        .          . |
 65. | 1.004242   .3877551 |
 66. | 1.004242   .3877551 |
 67. | 1.004242   .3877551 |
     |---------------------|
 68. | 1.004242   .3877551 |
 69. | 1.004242   .3877551 |
 70. | 1.004242   .3877551 |
 71. | 1.004242   .3877551 |
 72. | 1.004242   .3877551 |
     |---------------------|
 73. | 1.004242   .3877551 |
 74. | 1.004242   .3877551 |
     +---------------------+

.

[Paris Rira]

Wow it is perfect, you are helpful as always. Thank you so much.

Now the matter of disparity between entropyetc and hhi ( I understand that hhi is not written by you, in case you have any ideas please share them with me, thanks)
I think that the result of Simpson through entropyetc should be equal with hhi, though as the below example shows they are different (0.297 from entropyetc, and 0.156 from hhi)

 . entropyetc rep78, list

  +-----------------------------------------------------------+
  |       distinct   Shannon H   exp(H)   Simpson   1/Simpson |
  |-----------------------------------------------------------|
  | all          5       1.358    3.888     0.297       3.369 |
  +-----------------------------------------------------------+

. hhi  rep78

. sum hhi

    Variable |        Obs        Mean    Std. Dev.       Min        Max
-------------+---------------------------------------------------------
   hhi_rep78 |         74    .0156994           0   .0156994   .0156994

. tab hhi

  hhi_rep78 |      Freq.     Percent        Cum.
------------+-----------------------------------
   .0156994 |         74      100.00      100.00
------------+-----------------------------------
      Total |         74      100.00

[Paris Rira]

I have a question about gen. Why it does not support 5? because the last one 1/Simpson can't be generated while it is printed. At most, it is allowed to:

 entropyetc Educ if foreign == 1, by(NPC_FIC year) gen(1=A 2=B 3=C 4=D )

Rephrase with Ginger (Ctrl+Alt+E)

[Nick Cox]

#12 You found a bug. Here is the fixed code. Thanks for letting me know!

*! 3.0.1 NJC 17 June 2024
*! 3.0.0 NJC 11 January 2024
*! 2.0.2 NJC 29 June 2021
*! 2.0.1 NJC 15 June 2020
*! 2.0.0 NJC 5 July 2018
*! 1.0.0 NJC 20 November 2016
program entropyetc, sortpreserve    
    version 8.2
    syntax varname [if] [in] [aweight fweight/] ///
    [, GENerate(str)  by(varlist) list Format(str) * ]
    
    if "`generate'`list'" == "" {
        di "nothing to do"
        exit 0
    }

    quietly {
        marksample touse, strok  
        if "`by'" != "" markout `touse' `by', strok
        count if `touse'
        if r(N) == 0 error 2000
    
        if "`by'" == "" {
            tempvar by
            gen byte `by' = `touse'
            label def `by' 1 "all"
            label val `by' `by'
            char `by'[varname] " "
        }

        if "`generate'" != "" parsegenerate `generate'
    
        tempvar freq total S p Shannon Simpson Shannon2 Simpson2
    
        if "`exp'" == ""  local exp = 1
    
        bysort `touse' `by' `varlist' : gen double `freq' = sum(`touse' * `exp')  
        by `touse' `by' `varlist' : replace `freq' = cond(_n == _N, `freq'[_N], .)
        by `touse' `by' `varlist' : gen `S' = cond(_n == _N, `freq'[_N] > 0, .)
        bysort `touse' `by' (`S') : replace `S' = sum(`S')
        by `touse' `by' : replace `S' = `S'[_N]
        by `touse' `by' : gen double `total' = sum(`freq')
        by `touse' `by' : replace `total' = `total'[_N]
            
        gen double `p' = `freq' / `total'
    
        by `touse' `by': gen double `Shannon' = sum(`p' * ln(1/`p'))
        by `touse' `by': replace `Shannon' = `Shannon'[_N]
        gen double `Shannon2' = exp(`Shannon')
    
        by `touse' `by': gen double `Simpson' = sum(`p'^2)
        by `touse' `by': replace `Simpson' = `Simpson'[_N]
        gen double `Simpson2' = 1/`Simpson'
    
        label var `S'        "distinct"
        label var `Shannon'  "Shannon H"
        label var `Shannon2' "exp(H)"
        label var `Simpson'  "Simpson"
        label var `Simpson2' "1/Simpson"
            
        if "`format'" == "" local format "%4.3f"
        format `S' %1.0f
        format `Shannon' `Shannon2' `Simpson' `Simpson2' `format'
    }    

    tokenize `S' `Shannon' `Shannon2' `Simpson' `Simpson2'  
    
    if "`list'" != "" {
        forval j = 1/5 {
            char ``j''[varname] "`: var label ``j'''"
        }
    
        tempvar tolist
        egen `tolist' = tag(`touse' `by')
    
        list `by' `S' `Shannon' `Shannon2' `Simpson' `Simpson2' if `touse' & `tolist', ///
        abbrev(9) subvarname noobs `options'
    }    

    forval j = 1/5 {
        if "`var_`j''" != "" {
            gen `var_`j'' = ``j'' if `touse'
        }
    }

end

program parsegenerate
    tokenize `0'
    if "`6'" != "" {
        di as err "generate() should specify 1 to 5 tokens"
        exit 134
    }

    forval j = 1/5 {
        if "``j''" != "" {
            gettoken no rest : `j', parse(=)  
            capture numlist "`no'", max(1) int range(>=1 <=5)
            if _rc {
                di as err "generate() error: ``j''"
                exit _rc
            }

            gettoken eqs rest : rest, parse(=)
            confirm new var `rest'
            c_local var_`no' "`rest'"
        }
    }  
end

The line now

capture numlist "`no'", max(1) int range(>=1 <=5)

had 4 instead of 5.

[Nick Cox]

#11 On hhi from SSC (Muhammad Rashid Ansari)

See my comments made at #6 which pointed out a key difference between commands in this territory.

hhi wasn't written by me so I don't feel responsible for it. If its results puzzle you, then post a concrete example with a simple dataset. "I don't get the same results" is not a good problem statement.

But entropyetc treats the values of its main argument as defining the distinct levels of a categorical variable. That's quite different from any command that interprets them as giving the frequencies of those categories.

The difference is one reason why different commands exist in this territory.

Here is one demonstration of how the commands can be reconciled. It's a demonstration of how, as said, hhi treats its argument as frequencies or equivalent while by default entropyetc calculates those for you for distinct levels.

. sysuse auto, clear 
(1978 automobile data)

. 
. contract rep78 if rep78 < . 

. 
. hhi _freq 

. 
. list 

     +--------------------------+
     | rep78   _freq   hhi__f~q |
     |--------------------------|
  1. |     1       2   .2967864 |
  2. |     2       8   .2967864 |
  3. |     3      30   .2967864 |
  4. |     4      18   .2967864 |
  5. |     5      11   .2967864 |
     +--------------------------+

. 
. entropyetc rep78 [fw=_freq], list 

  +-----------------------------------------------------------+
  |       distinct   Shannon H   exp(H)   Simpson   1/Simpson |
  |-----------------------------------------------------------|
  | all          5       1.358    3.888     0.297       3.369 |
  +-----------------------------------------------------------+

[Paris Rira]

Prof Nick,
Thank you so much for the explanation, I fully got the point of disparity.