Announcement

Collapse
No announcement yet.
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • clustered heatmap in Stata

    Many children, when they see something they like, will wave their little hands at their mom and dad and say, "I want it, I want it." Just like those children, when I see some statistical graphs in R or Python, I will also use my hands to search or help at Stata command window, saying to Stata, "I want it, I want it." Today, while looking at the correlation matrix produced by -tetrachoric- and the -heatplot- command written by Ben Jann, I was wondering: do we have—or could we have—something like pheatplot from R? pheatplot is clustered heatmaps; of course, the "p" in its name stands for "pretty"—the author considers it a pretty heatmap, and indeed it lives up to the name.

    Pheatmap: Beautiful heatmaps, written in one line of R.

    pheatmap function in R

    A little help with Heatmap and commands related
    Click image for larger version

Name:	hero-heatmap-B3fDCmEL.png
Views:	1
Size:	988.4 KB
ID:	1788101


  • #2
    Data example (heatplot is from SSC/Github). There's -pheatplot- (requires Stata 17+) on SSC, but it is Visualizing statistical significance across coefficients.

    Code:
    webuse familyvalues
    tetrachoric RS056 RS057 RS058 RS059 RS060 RS061 RS062 RS063 RS064 RS065 RS066 RS067 RS068 RS069 RS070 RS071 RS072 RS073 RS074 RS075 RS076
    matrix F=r(Rho)
    heatplot F, xlabel(, angle(45))
    Click image for larger version

Name:	Graph.png
Views:	1
Size:	284.3 KB
ID:	1788104

    Comment


    • #3
      Chen, your example prompted me to try a small implementation using only official Stata commands and Mata.
      Click image for larger version

Name:	clustered_heatmap.png
Views:	1
Size:	355.1 KB
ID:	1788111

      The code below downloads the familyvalues example data, estimates the tetrachoric correlations, and draws a clustered heatmap with a dendrogram above it. No additional packages or attached files are required. Please run the entire block together from the Do-file Editor.

      For clustering, I use dissimilarity = 1 − rho and average linkage. This groups strongly positively correlated variables together. Both axes use the same dendrogram leaf order, while the colors display the original correlations.

      Code:
      version 16
      set more off
      preserve
      
      * 1. Obtain the data and tetrachoric correlation matrix
      webuse familyvalues, clear
      local vars RS056 RS057 RS058 RS059 RS060 RS061 RS062 RS063 ///
          RS064 RS065 RS066 RS067 RS068 RS069 RS070 RS071
      
      quietly tetrachoric `vars'
      matrix F = r(Rho)
      
      mata: assert(!hasmissing(st_matrix("F")))
      mata: assert(max(abs(st_matrix("F")-st_matrix("F")'))<1e-10)
      mata: st_matrix("D",J(rows(st_matrix("F")),cols(st_matrix("F")),1)-st_matrix("F"))
      
      * 2. Average-linkage clustering of D = 1 - rho
      clustermat averagelinkage D, name(hc) clear
      sort hc_id
      
      * Reorder the matrix and construct dendrogram segments
      mata:
      F=st_matrix("F")
      o=st_data(.,"hc_ord")
      h=st_data(.,"hc_hgt")
      n=rows(o)
      R=F[o,o]
      st_matrix("Rordered",R)
      
      g=(1::n)
      xc=g
      ht=J(n,1,0)
      seg=J(3*(n-1),4,.)
      b=order(h[1..n-1],1)
      
      for(k=1;k<n;k++) {
          j=b[k]
          a=g[j]
          c=g[j+1]
          assert(a!=c)
      
          seg[3*k-2,.]=(xc[a],ht[a],xc[a],h[j])
          seg[3*k-1,.]=(xc[c],ht[c],xc[c],h[j])
          seg[3*k,.]=(xc[a],h[j],xc[c],h[j])
      
          xc[a]=(xc[a]+xc[c])/2
          ht[a]=h[j]
          for(t=1;t<=n;t++) if(g[t]==c) g[t]=a
      }
      
      assert(rows(uniqrows(g))==1)
      st_matrix("Segments",seg)
      st_numscalar("Hmax",max(h[1..n-1]))
      end
      
      * Axis labels in dendrogram leaf order
      local xl
      local yl
      forvalues i=1/16 {
          local id=hc_ord[`i']
          local v : word `id' of `vars'
          local xl `xl' `i' "`v'"
          local yy=17-`i'
          local yl `yl' `yy' "`v'"
      }
      
      * 3. Construct the 16 x 16 heatmap
      clear
      set obs 256
      generate double xx=mod(_n-1,16)+1
      generate double yy=16-floor((_n-1)/16)
      generate double rho=.
      
      forvalues i=1/16 {
          forvalues j=1/16 {
              local row=(`i'-1)*16+`j'
              replace rho=Rordered[`i',`j'] in `row'
          }
      }
      
      generate double lo=yy-.5
      generate double hi=yy+.5
      generate int bin=min(20,max(0,round((rho+1)*10)))
      
      * Add a color key using the same 21 color levels
      set obs 277
      forvalues k=0/20 {
          local r=257+`k'
          replace xx=18 in `r'
          replace lo=.5+`k'*16/21 in `r'
          replace hi=.5+(`k'+1)*16/21 in `r'
          replace bin=`k' in `r'
      }
      
      * Place the dendrogram above the heatmap
      svmat double Segments, names(s)
      replace s2=17.3+3*s2/Hmax
      replace s4=17.3+3*s4/Hmax
      
      * Blue-white-red palette
      local plots
      forvalues k=0/20 {
          local t=abs(`k'-10)/10
          if `k'<=10 {
              local rr=round(247+(33-247)*`t')
              local gg=round(247+(102-247)*`t')
              local bb=round(247+(172-247)*`t')
          }
          else {
              local rr=round(247+(178-247)*`t')
              local gg=round(247+(24-247)*`t')
              local bb=round(247+(43-247)*`t')
          }
          local plots `plots' ///
              (rbar lo hi xx if bin==`k', barwidth(1) ///
              fcolor("`rr' `gg' `bb'") fintensity(100) lwidth(none))
      }
      
      * 4. Draw everything in the same coordinate system
      twoway `plots' ///
          (pcspike s2 s1 s4 s3, lcolor(gs4) lwidth(thin)), ///
          xlabel(`xl', angle(45) labsize(small) noticks nogrid) ///
          ylabel(`yl', angle(0) labsize(small) noticks nogrid) ///
          xscale(range(.5 20) noline) ///
          yscale(range(.5 21.6) noline) ///
          xtitle("") ytitle("") legend(off) ///
          title("Family values: clustered tetrachoric correlations", ///
              size(medsmall)) ///
          subtitle("Average linkage | dissimilarity = 1 - rho", ///
              size(small)) ///
          text(17.1 18 "rho", size(small)) ///
          text(.9 19.1 "-1.0" 4.5 19.1 "-0.5" 8.5 19.1 "0" ///
              12.5 19.1 "0.5" 16.1 19.1 "1.0", size(vsmall)) ///
          text(21.1 8 ///
              "Tree height: dissimilarity (top merge = `=string(Hmax,"%4.2f")')", ///
              size(vsmall)) ///
          note("Same leaf order on both axes. Blue: negative; red: positive." ///
              "Source: Stata Press familyvalues. Branch heights show dissimilarity.", ///
              size(vsmall)) ///
          graphregion(color(white)) ///
          plotregion(color(white) margin(zero)) ///
          aspectratio(1.082) xsize(9) ysize(10) ///
          name(clustered_heatmap, replace)
      
      restore
      The tree and heatmap share their horizontal coordinates, so each dendrogram leaf aligns with the corresponding column. The branch heights are proportionally rescaled to fit above the matrix. The heatmap uses 21 color levels spanning −1 to 1.

      One methodological distinction is worth making: clustering on 1 − rho is different from clustering the rows of the correlation matrix by the similarity of their correlation profiles. This example implements the former; it does not attempt to reproduce R’s pheatmap defaults.

      I tested this in Stata/MP 16.1 and checked the tree against Stata’s official dendrogram output. It is a demonstration tailored to these 16 variables, rather than a general pheatmap replacement, but it shows that the combined display is feasible within Stata.

      Comment


      • #4
        The gene-expression example in Elio’s earlier thread illustrates a slightly different application from the correlation-matrix example above.
        In that case, we would retain genes as rows and individual subjects as columns, cluster the two dimensions separately, and use a color strip to identify each subject’s group. This preserves individual observations rather than reducing each gene–group combination to a single mean or median, addressing the concern Nick raised about within-group variation.
        The following self-contained example uses synthetic data for 35 subjects, seven genes, and three groups. It does not reconstruct Elio’s observations. Group differences are deliberately introduced into the simulation to illustrate the display.
        For real data, we would need a unique subject identifier, group membership, and complete measurements for the seven genes. Here, each gene is standardized across all subjects, not separately within groups. Group membership is used only for the annotation strip, not for clustering.
        The code uses Euclidean distance and average linkage for both dimensions. It requires no additional packages and was tested in Stata/MP 16.1. Please run the whole block together from the Do-file Editor.
        Click image for larger version

Name:	elio_heatmap.png
Views:	1
Size:	183.3 KB
ID:	1788113

        Code:
        version 16
        set more off
        preserve
        clear
        
        * 1. Generate SYNTHETIC data, not Elio's observations
        set seed 20261114
        set obs 35
        generate int CODE=_n
        generate byte group=cond(_n<=12,1,cond(_n<=24,2,3))
        label define elio_group 1 "HC" 2 "BF-" 3 "BF+", replace
        label values group elio_group
        
        local genes IFNALPHAmRNA IFNAR1mRNA IFNAR2mRNA ///
            IFNBETAmRNA ISG15mRNA ISG56mRNA UNC93B1mRNA
        
        generate double latent=rnormal()
        local j=0
        foreach gene of local genes {
            local ++j
            generate double raw`j'=.45*latent+rnormal()+ ///
                cond(`j'<=3,.8*(group-2),cond(`j'<=5,-.6*(group-2),0))
            egen double z_`gene'=std(raw`j')
        }
        drop raw* latent
        
        * Real data would enter here in wide form:
        * one row per CODE, numeric group, and seven z_ variables.
        * The drawing dimensions below are fixed at 35 subjects and 7 genes.
        
        local zs
        foreach gene of local genes {
            local zs `zs' z_`gene'
        }
        isid CODE
        assert inrange(group,1,3)
        foreach v of local zs {
            assert !missing(`v')
        }
        mkmat `zs', matrix(Z)
        mkmat CODE group, matrix(IDs)
        
        * 2. Utilities for distances and dendrogram coordinates
        capture mata: mata drop elio_distance()
        capture mata: mata drop elio_segments()
        
        mata:
        real matrix elio_distance(real matrix X) {
            real scalar i,j,n
            real matrix D
            n=rows(X)
            D=J(n,n,0)
            for(i=1;i<n;i++) for(j=i+1;j<=n;j++) {
                D[i,j]=sqrt(sum((X[i,.]-X[j,.]):^2))
                D[j,i]=D[i,j]
            }
            return(D)
        }
        
        real matrix elio_segments(real colvector h) {
            real scalar n,k,j,a,c,t
            real colvector g,xc,ht,b
            real matrix s
            n=rows(h)
            g=(1::n)
            xc=g
            ht=J(n,1,0)
            s=J(3*(n-1),4,.)
            b=order(h[1..n-1],1)
        
            for(k=1;k<n;k++) {
                j=b[k]
                a=g[j]
                c=g[j+1]
                assert(a!=c)
                s[3*k-2,.]=(xc[a],ht[a],xc[a],h[j])
                s[3*k-1,.]=(xc[c],ht[c],xc[c],h[j])
                s[3*k,.]=(xc[a],h[j],xc[c],h[j])
                xc[a]=(xc[a]+xc[c])/2
                ht[a]=h[j]
                for(t=1;t<=n;t++) if(g[t]==c) g[t]=a
            }
            return(s)
        }
        
        st_matrix("Ds",elio_distance(st_matrix("Z")))
        st_matrix("Dg",elio_distance(st_matrix("Z")'))
        end
        
        * 3. Cluster subjects and genes independently
        clustermat averagelinkage Ds, name(subj) clear
        sort subj_id
        mata: st_matrix("OS",st_data(.,"subj_ord"))
        mata: st_matrix("TS",elio_segments(st_data(.,"subj_hgt")))
        
        clustermat averagelinkage Dg, name(gene) clear
        sort gene_id
        mata: st_matrix("OG",st_data(.,"gene_ord"))
        mata: st_matrix("TG",elio_segments(st_data(.,"gene_hgt")))
        
        mata:
        Z=st_matrix("Z")
        os=st_matrix("OS")
        og=st_matrix("OG")
        R=Z[os,og]'
        st_matrix("R",R)
        st_numscalar("Cmax",ceil(max(abs(R))))
        assert(rows(R)==7 & cols(R)==35)
        
        S=st_matrix("TS")
        G=st_matrix("TG")
        
        // Top dendrogram
        S[,2]=8.8:+2.5*S[,2]/max(S[,2])
        S[,4]=8.8:+2.5*S[,4]/max(st_matrix("TS")[,4])
        
        // Left dendrogram
        T=J(rows(G),4,.)
        T[,1]=-5*G[,2]/max(G[,4])
        T[,2]=8:-G[,1]
        T[,3]=-5*G[,4]/max(G[,4])
        T[,4]=8:-G[,3]
        st_matrix("TREE",S\T)
        end
        
        * Labels follow the appropriate leaf orders
        local xl
        forvalues j=1/35 {
            local orig=OS[`j',1]
            local id=IDs[`orig',1]
            local xl `xl' `j' "S`id'"
        }
        
        local glabs
        forvalues i=1/7 {
            local orig=OG[`i',1]
            local gene : word `orig' of `genes'
            local y=8-`i'
            local glabs `glabs' `y' 36 "`gene'"
        }
        
        * 4. Construct heatmap cells
        clear
        set obs 245
        generate double x=mod(_n-1,35)+1
        generate double y=7-floor((_n-1)/35)
        generate double value=.
        
        forvalues i=1/7 {
            forvalues j=1/35 {
                local r=(`i'-1)*35+`j'
                replace value=R[`i',`j'] in `r'
            }
        }
        
        generate double low=y-.5
        generate double high=y+.5
        generate int bin=round((value+Cmax)/(2*Cmax)*30)
        assert inrange(bin,0,30)
        
        * Group strip follows the reordered subjects
        set obs 311
        generate byte grp=.
        forvalues j=1/35 {
            local r=245+`j'
            local orig=OS[`j',1]
            replace x=`j' in `r'
            replace low=8 in `r'
            replace high=8.4 in `r'
            replace grp=IDs[`orig',2] in `r'
        }
        
        * Color key
        forvalues k=0/30 {
            local r=281+`k'
            replace x=43 in `r'
            replace low=.5+`k'*7/31 in `r'
            replace high=.5+(`k'+1)*7/31 in `r'
            replace bin=`k' in `r'
        }
        svmat double TREE, names(t)
        
        * Blue-white-red palette
        local plots
        forvalues k=0/30 {
            local q=abs(`k'-15)/15
            if `k'<=15 {
                local rr=round(247+(33-247)*`q')
                local gg=round(247+(102-247)*`q')
                local bb=round(247+(172-247)*`q')
            }
            else {
                local rr=round(247+(178-247)*`q')
                local gg=round(247+(24-247)*`q')
                local bb=round(247+(43-247)*`q')
            }
            local plots `plots' ///
                (rbar low high x if bin==`k', barwidth(1) ///
                fcolor("`rr' `gg' `bb'") fintensity(100) lwidth(none))
        }
        
        * 5. Draw heatmap, group strip and both dendrograms
        twoway `plots' ///
            (rbar low high x if grp==1, barwidth(1) ///
                color("0 140 140") fintensity(100) lwidth(none)) ///
            (rbar low high x if grp==2, barwidth(1) ///
                color("210 178 105") fintensity(100) lwidth(none)) ///
            (rbar low high x if grp==3, barwidth(1) ///
                color("125 90 160") fintensity(100) lwidth(none)) ///
            (pcspike t2 t1 t4 t3, lcolor(gs4) lwidth(thin)), ///
            xlabel(`xl',angle(90) labsize(vsmall) noticks nogrid) ///
            ylabel(none) ///
            xscale(range(-5.5 46) noline) ///
            yscale(range(.5 11.8) noline) ///
            xtitle("") ytitle("") ///
            text(`glabs',place(e) size(vsmall)) ///
            text(8 43 "z-score",size(vsmall)) ///
            text(.7 45 "-`=Cmax'" 4 45 "0" 7.3 45 "`=Cmax'", ///
                size(vsmall)) ///
            title("Gene expression: subjects and genes clustered separately", ///
                size(medsmall)) ///
            subtitle("SYNTHETIC DATA | 35 subjects, 7 genes | Euclidean distance, average linkage", ///
                size(small)) ///
            legend(order(32 "HC" 33 "BF-" 34 "BF+") rows(1) ///
                size(small) region(lstyle(none))) ///
            note("Each column is one subject; the strip marks group membership. No group averaging." ///
                "Gene-wise z-scores across all 35 subjects. Branch lengths rescaled separately on each axis.", ///
                size(vsmall)) ///
            graphregion(color(white)) ///
            plotregion(color(white) margin(zero)) ///
            xsize(14) ysize(7) name(elio_heatmap,replace)
        
        restore
        This produces the gene-by-subject display discussed in the earlier thread while retaining individual variation. The top strip also makes it possible to see whether subjects from the same known group tend to cluster together, without forcing them into adjacent columns.
        The two dendrograms are rescaled independently for display, so their drawn branch lengths should not be compared numerically across axes. Also, this is an exploratory visualization: apparent separation in a heatmap does not itself establish statistically significant group differences.
        The example is deliberately tailored to 35 subjects and seven genes. Applying it to Elio’s complete data would require retaining CODE and the full measurements; appropriately standardized z-scores should not be standardized again merely to run the plot.

        Comment


        • #5
          R's code:

          Code:
          library(haven)
          library(dplyr)
          library(psych)
          library(pheatmap)
          
          # 1. read RS056–RS076
          dat <- read_dta("https://www.stata-press.com/data/r16/familyvalues.dta")
          bin_vars <- paste0("RS0", 56:76)
          mat <- dat %>%
            select(all_of(bin_vars)) %>%
            mutate(across(everything(), as.numeric)) %>%
            as.matrix()
          
          # 2. caculate tetrachoric
          tet <- tetrachoric(mat, na.rm = TRUE)
          cor_mat <- tet$rho
          
          # 3. plot heatmap
          pheatmap(
            cor_mat,
            cluster_rows = TRUE,
            cluster_cols = TRUE,
            color = hcl.colors(100, "BluYl"),
            breaks = seq(-1, 1, length.out = 101),
            main = "Tetrachoric correlation: RS056–RS076",
            fontsize = 9
          )
          Click image for larger version

Name:	Rplot02.png
Views:	1
Size:	15.2 KB
ID:	1788115

          Comment


          • #6
            Dear Choonjoo Lee, Thank you for putting this together. I guess you will reply. It is an impressive demonstration of what can be achieved using only official Stata commands and Mata. Using rbar to simulate heatmap cells and pcspike for the dendrogram is a so clever workaround that produces a publication-quality figure.

            I particularly appreciate your note on the methodological distinction between clustering on dissimilarity (1 − rho) versus clustering based on the similarity of correlation profiles. This is a nuance often overlooked when people migrate from pheatmap in R, where the default clustering operates on the rows of the matrix using Euclidean distance.

            Code:
            library(haven)
            library(dplyr)
            library(psych)
            library(pheatmap)
            
            # 1. read RS056–RS076
            dat <- read_dta("https://www.stata-press.com/data/r16/familyvalues.dta")
            bin_vars <- paste0("RS0", 56:76)
            mat <- dat %>%
                select(all_of(bin_vars)) %>%
                mutate(across(everything(), as.numeric)) %>%
                as.matrix()
            
            # 2. caculate tetrachoric
            tet <- tetrachoric(mat, na.rm = TRUE)
            cor_mat <- tet$rho
            
            # 3. plot heatmap
            pheatmap(
                cor_mat,
                cluster_rows = TRUE,
                cluster_cols = TRUE,
                color = colorRampPalette(c("navy", "white", "firebrick3"))(100),
                breaks = seq(-1, 1, length.out = 101),
                main = "Tetrachoric correlation: RS056–RS076",
                fontsize = 9
            )
            Click image for larger version

Name:	Rplot03.png
Views:	1
Size:	15.2 KB
ID:	1788117

            Comment


            • #7
              Thank you, Chen, for sharing the R code. It makes the comparison much clearer.
              Click image for larger version

Name:	chen_followup.png
Views:	1
Size:	272.7 KB
ID:	1788119

              Your example uses 21 variables, RS056–RS076. Also, pheatmap defaults to Euclidean distances between the rows and columns of the supplied matrix, followed by complete-linkage clustering. My first Stata example used 16 variables, dissimilarity = 1 − rho, and average linkage.
              The following version uses your variable range and clustering choices. Thus, it groups variables by the similarity of their correlation profiles, rather than using 1 − rho directly as the dissimilarity. It also includes dendrograms on both axes.
              The displayed variable order may still differ from your figure. I checked this by passing the same Stata-estimated correlation matrix to R’s hclust with Euclidean distance and complete linkage. The leaf orders differed, but the pairwise merge heights agreed to numerical precision: the maximum absolute difference was approximately 4.4 × 10^−16. Different left–right rotations of branches can therefore change the displayed order without changing the underlying clustering.
              This check isolates the clustering step. It does not establish exact agreement with your complete R workflow, because Stata’s tetrachoric and psych::tetrachoric may produce different correlation matrices under their respective defaults. An exact comparison would require the same correlation matrix in both implementations.
              The code below retains the blue–white–red palette from my earlier example. It uses only official Stata commands and Mata and was tested in Stata/MP 16.1. Please run the entire block together from the Do-file Editor.
              Code:
              version 16
              set more off
              preserve
              
              * 1. Estimate correlations for RS056-RS076
              webuse familyvalues, clear
              local vars
              forvalues k=56/76 {
                  local vars `vars' RS0`k'
              }
              quietly tetrachoric `vars'
              matrix F=r(Rho)
              local n=rowsof(F)
              
              * 2. Euclidean distances between correlation profiles
              mata:
              F=st_matrix("F")
              assert(!hasmissing(F))
              n=rows(F)
              D=J(n,n,0)
              for(i=1;i<n;i++) for(j=i+1;j<=n;j++) {
                  D[i,j]=sqrt(sum((F[i,.]-F[j,.]):^2))
                  D[j,i]=D[i,j]
              }
              st_matrix("D",D)
              end
              
              clustermat completelinkage D, name(hc) clear
              sort hc_id
              
              * 3. Reorder the matrix and construct dendrogram segments
              mata:
              F=st_matrix("F")
              o=st_data(.,"hc_ord")
              h=st_data(.,"hc_hgt")
              n=rows(o)
              st_matrix("R",F[o,o])
              
              g=(1::n)
              xc=g
              ht=J(n,1,0)
              S=J(3*(n-1),4,.)
              b=order(h[1..n-1],1)
              
              for(k=1;k<n;k++) {
                  j=b[k]
                  a=g[j]
                  c=g[j+1]
                  assert(a!=c)
                  S[3*k-2,.]=(xc[a],ht[a],xc[a],h[j])
                  S[3*k-1,.]=(xc[c],ht[c],xc[c],h[j])
                  S[3*k,.]=(xc[a],h[j],xc[c],h[j])
                  xc[a]=(xc[a]+xc[c])/2
                  ht[a]=h[j]
                  for(t=1;t<=n;t++) if(g[t]==c) g[t]=a
              }
              
              H=max(h[1..n-1])
              T=S
              L=S
              
              // Top dendrogram
              T[,2]=(n+1.3):+3*S[,2]/H
              T[,4]=(n+1.3):+3*S[,4]/H
              
              // Left dendrogram
              L[,1]=-3*S[,2]/H
              L[,2]=(n+1):-S[,1]
              L[,3]=-3*S[,4]/H
              L[,4]=(n+1):-S[,3]
              
              st_matrix("TREE",T\L)
              end
              
              * Labels follow Stata's dendrogram leaf order
              local xl
              local labs
              forvalues i=1/`n' {
                  local id=hc_ord[`i']
                  local v : word `id' of `vars'
                  local xl `xl' `i' "`v'"
                  local y=`n'+1-`i'
                  local labs `labs' `y' `=`n'+1' "`v'"
              }
              
              * 4. Heatmap cells
              clear
              set obs `=`n'^2'
              generate double x=mod(_n-1,`n')+1
              generate double y=`n'-floor((_n-1)/`n')
              generate double rho=.
              
              forvalues i=1/`n' {
                  forvalues j=1/`n' {
                      local r=(`i'-1)*`n'+`j'
                      quietly replace rho=R[`i',`j'] in `r'
                  }
              }
              
              generate double lo=y-.5
              generate double hi=y+.5
              generate byte bin=min(20,max(0,round((rho+1)*10)))
              
              * Color key
              set obs `=`n'^2+21'
              forvalues k=0/20 {
                  local r=`n'^2+1+`k'
                  quietly replace x=`n'+5 in `r'
                  quietly replace lo=.5+`k'*`n'/21 in `r'
                  quietly replace hi=.5+(`k'+1)*`n'/21 in `r'
                  quietly replace bin=`k' in `r'
              }
              svmat double TREE, names(t)
              
              * Blue-white-red palette
              local plots
              forvalues k=0/20 {
                  local q=abs(`k'-10)/10
                  if `k'<=10 {
                      local rr=round(247+(33-247)*`q')
                      local gg=round(247+(102-247)*`q')
                      local bb=round(247+(172-247)*`q')
                  }
                  else {
                      local rr=round(247+(178-247)*`q')
                      local gg=round(247+(24-247)*`q')
                      local bb=round(247+(43-247)*`q')
                  }
                  local plots `plots' (rbar lo hi x if bin==`k', ///
                      barwidth(1) fcolor("`rr' `gg' `bb'") ///
                      fintensity(100) lwidth(none))
              }
              
              * 5. Integrated heatmap and dendrograms
              twoway `plots' ///
                  (pcspike t2 t1 t4 t3, lcolor(gs4) lwidth(thin)), ///
                  xlabel(`xl',angle(90) labsize(vsmall) noticks nogrid) ///
                  ylabel(none) ///
                  xscale(range(-3.5 `=`n'+7') noline) ///
                  yscale(range(.5 `=`n'+5') noline) ///
                  xtitle("") ytitle("") legend(off) ///
                  text(`labs',place(e) size(vsmall)) ///
                  text(`=`n'+1.2' `=`n'+5' "rho",size(vsmall)) ///
                  text(.8 `=`n'+6.3' "-1" ///
                      `=(`n'+1)/2' `=`n'+6.3' "0" ///
                      `=`n'+.2' `=`n'+6.3' "1",size(vsmall)) ///
                  title("Tetrachoric correlations: RS056-RS076", ///
                      size(medsmall)) ///
                  subtitle("Euclidean distance between correlation profiles | complete linkage", ///
                      size(small)) ///
                  note("Correlations estimated in Stata. Blue: negative; red: positive." ///
                      "Both axes share the same tree; branch heights are proportionally rescaled.", ///
                      size(vsmall)) ///
                  graphregion(color(white)) ///
                  plotregion(color(white) margin(zero)) ///
                  xsize(10) ysize(9) name(chen_followup,replace)
              
              restore
              Because the correlation matrix is symmetric, its row and column distance matrices are identical. This implementation therefore computes one tree and uses it on both axes, keeping the matrix entries, labels, and dendrogram leaves aligned.
              The example matches the variable range, distance definition, and linkage method in your R code. Exact visual reproduction would additionally require matching the estimated correlation matrix, branch orientation, and graphical settings.

              Comment


              • #8
                Dear Choonjoo Lee Thank you for this incredibly thorough and rigorous follow-up. I am impressed by the precision of your comparative check. It perfectly demonstrates that clustermat completelinkage (on Euclidean distances of the profiles) in Stata and R's default pheatmap setup are mathematically doing the exact same thing.

                Below is cor_mat returned by psych::tetrachoric()

                Code:
                . matrix list R_Tetrachoric
                
                symmetric R_Tetrachoric[21,21]
                             c1          c2          c3          c4          c5          c6          c7
                 r1           1
                 r2   .21523615           1
                 r3    .3714883   .04079048           1
                 r4   .28665152   .01028496   .40589642           1
                 r5  -.15286072  -.08024234   -.0623422   .02658193           1
                 r6  -.19168716   .09964547  -.23925537    -.359148   .11410286           1
                 r7    .3119887   .03772048   .36130317   .61311214  -.12577085  -.48247469           1
                 r8   .32532221   .22926737   .17216827   .14693253   .03826033  -.04577857    .1597528
                 r9    .1957484   .18370737   .12566064   .15722797   .07312603  -.06343672   .15776071
                r10  -.21955512  -.13732892  -.02873423  -.06585619    .2563734   .05552264  -.09854684
                r11   .16911376   .05391472   .25274213   .22999862  -.05027042  -.23219656   .33617358
                r12   .16182863    .1209234   .16192251   .23476579  -.05995767   -.1515578   .33273939
                r13   .20896168  -.01138054   .28350175   .35053935  -.10674575  -.28054148   .45474328
                r14   .07610844  -.06438478   .22228667   .25779177   .12529631  -.21583448   .26642514
                r15  -.16770047  -.00704632  -.08422779  -.09703935   .42290462    .1472661  -.28186449
                r16   .10575991   .37531854  -.04423225  -.00354021  -.04400034   .15408565    .0364018
                r17   .16601506   .31890222   .04933777   .03382385   .01666005    .0807824   .10247581
                r18   .03000584   .05129923   .08744868   .06924337     .111274  -.04697823   .10316956
                r19   .18218457   .19018694   .18587361   .12542694   .05302842  -.08688475   .20328019
                r20   .24415051  -.06333789   .40756903   .42073095  -.04505562  -.53108655   .56505953
                r21   .35849615   .03158695   .42430613   .46787007    -.084113   -.4181491    .6092839
                
                             c8          c9         c10         c11         c12         c13         c14
                 r8           1
                 r9   .34507471           1
                r10  -.08403268  -.00830072           1
                r11   .07379799    .0988417  -.04150015           1
                r12   .14861881   .16479345  -.11488604   .16136699           1
                r13   .04533405   .06948659  -.03902143   .28676574   .22890054           1
                r14   .04721097    .1244223   .01419202    .1635744   .18296445   .46847688           1
                r15   .04726608   .05518542   .22351236  -.14332865  -.02501318  -.13926658   .16314003
                r16   .20590529   .26146259  -.11040513  -.08335171   .19913632  -.02416496   -.0292498
                r17   .21537405   .21009541  -.11984463  -.03767856   .23502477  -.01585095   .00890552
                r18   .16115384   .13025687   .01625179  -.04436614    .2219675   .10587953   .24725907
                r19    .2642896   .24629995  -.06070408    .0992814   .21488899     .134554   .11551102
                r20   .08255317   .10807755  -.10674554   .39311729   .16483892   .42529271   .30318728
                r21   .15772188   .08090767  -.14804945     .340801   .25435454   .39779216   .31621674
                
                            c15         c16         c17         c18         c19         c20         c21
                r15           1
                r16   .12447344           1
                r17   .09857045   .60853379           1
                r18   .10432253   .31093515    .4141271           1
                r19   .07122803   .22019648   .29949505   .28760784           1
                r20  -.20511062  -.15646088  -.04554118   .03839547      .06787           1
                r21  -.22115708   .00131207   .07937639   .12335461    .2482844   .64281269           1

                Comment


                • #9
                  Dear Chen,

                  Thank you for posting the correlation matrix. This makes a direct comparison possible.

                  Using the matrix as printed in your post, I reproduced the variable order in your R figure with Euclidean distances between correlation profiles and complete linkage. Applying the same procedure in Stata produced the same clustering structure and merge heights: the maximum difference in pairwise cophenetic distances was approximately 6.7 × 10⁻¹⁶.

                  This confirms that the clustering calculations agree when the input matrix is identical. The earlier comparison involved slightly different estimated correlation matrices, as well as potentially different branch orientations. Your matrix therefore provides the missing piece for reproducing the heatmap in Stata.

                  Comment

                  Working...
                  X