Announcement

Collapse
No announcement yet.
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Getting datetime variables from strings - unable to replace, can only generate

    Background:
    -I am a Stata novice/improver, I have learnt to import from excel, drop variables, re-shape data from long form to wide form, de-string numeric variables, and run metan and metareg.
    -I am completing a meta-analysis where some outcomes are reported as times (e.g. time of sleep onset). I therefore have time variables which Stata interprets as strings (28 in total, including a mean and SD for an intervention group and a control group for each outcome).

    I am hoping there is a way to change / replace / recode time variables (which are currently strings) so that each one retains the same name, but is recognised as a I time. I think I am missing something basic. To explain tat is below, I decided to use a naming convention so that I can stay orientated to my data - such that mean sleep onset time in the intervention group is named: Ix_SOT_M. The only way I have been change these from strings to times so far is with four steps for each variable as follows:

    gen double temp = clock(Ix_SOT_M, "hm")
    drop Ix_SOT_M
    rename temp Ix_SOT_M
    format Ix_SOT_M %tc

    I tried but was incorrect with a few options including:
    . replace Ix_SOT_M = clock(Ix_SOT_M, "hm")
    type mismatch
    r(109);

    I am hoping there is a way to do this in fewer steps ?or even for all 28 variables using one process?
    If this is the most efficient way in terms of commands does anyone have any advice on how to best generate lots of similar commands but with a different variable name in each one, and how to then efficiently run these?

    Advice much appreciated.

    Sophie

  • #2
    clock() would produce numeric values for a numeric variable and all the advice is that you should use a double for such a variable. As lx_SOT_M is already string, you can't replace it with a numeric variable. Stata won't allow that and the main reason why is that most such requests are based on confusion. Yours is reasonable, but the rules remain as they are.

    What you can do is that something like this

    Code:
    gen double work w_Ix_SOT_M = clock(Ix_SOT_M, "hm")
    * check that it really is what you want
    drop lx_SOT_M
    rename w_lx_SOT_M lx_SOT_M
    where the checking step really is important.

    Looping is the answer to your general question.

    Comment


    • #3
      Hi Nick,

      Thank you very much for your response. So it looks like what I was doing was correct but inefficient.

      I have tried to understand the Stata help regarding loops but it seems a few steps beyond my current level of understanding, I thought maybe it should be the foreach command as it is variables I would need to loop over, but any attempts to apply the code resulted in invalid syntax messages. I don't suppose there are any videos of looping being applied to this type of problem? Or that it would be possible to explain how the looping code is applied to the variable names? I think part of my problem is not being able to understand whereabouts to insert my variable names and my commands within what is shown regarding loops.

      for instance for the first step I tried:

      foreach var of varlist Ix_SOT_M Ix_SOT_SD Cx_SOT_M {

      gen `var' = work()

      }


      Many Thanks,
      Sophie

      Comment


      • #4
        Strictly, it was an attempt to be efficient but was incorrect! Conversely, my code in #2 was wrong as work was illegal as well as superfluous. Sorry about that.

        I don't follow videos on programming to know what there is and I can't myself imagine learning programming from videos.

        I have written tutorial articles and FAQs on looping which something like

        search foreach
        should find

        I don't see where the checks I urged would come within your imagined code, but you may be seeking something like

        Code:
        foreach var of varlist Ix_SOT_M Ix_SOT_SD Cx_SOT_M { 
           gen double work = clock(`v', "hm")
           drop `v'
           rename work `v' 
        }
        -- except that this is dangerous unless you are sure that it does what you want.

        Comment


        • #5
          Hi, thank you again.

          I suppose I was thinking I could check the variables produced against my original data spreadsheet which is in excel. I have tried the code you suggested but I am just getting 'invalid syntax', should I have installed a custom package to be able to run this? Is there something I should be substituting for something else which I haven't understood?

          I have noticed that actually some of the variables were already correctly imported as times, whilst other variables were interpreted as strings, so I have been looking at the formatting within the original data sheet in excel to see if there is anything there I can alter so that the variables are correctly interpreted as times as soon as I import it. It seems that the way Stata has interpreted the variables is unconnected to the format they were in in excel however. I tried altering the format to 'time format' in excel for some variables and re-importing but Stata interpreted the same variables the same way even if the format in excel was changed. Is there anything to suggest here? I tried searching for advice on importing from excel and found archived advice here, but which appears to be out of date as I was able to change the format in excel without visual basic but it didn't alter the behaviour of stata: https://www.stata.com/statalist/arch.../msg01409.html

          Many Thanks
          Sophie

          Comment


          • #6
            My code should start

            Code:
             
             foreach v of varlist  Ix_SOT_M Ix_SOT_SD Cx_SOT_M {
            That was my fault again (sorry) -- but it's also true that you just provided a data example as requested in FAQ Advice #12 then (a) more people would have jumped on the question (b) we would have quickly checked that the code worked against real(istic) data.

            Comment


            • #7
              After a delay I have tried to get a data example out. I think perhaps this has not worked as desired though, this is not how the data looks to me on Stata in the data editor. If this displays correctly to you then maybe you can see how different variables have come in as different types of data, weirdly the same thing happens irrespective of what I alter in the format in excel so I assume Stata is deciding what data type to make a variable based on something other than the excel formatting.

              Code:
              * Example generated by -dataex-. To install: ssc install dataex
              clear
              input str5 Ix_SOT_M str8 Ix_SOT_SD double Ix_selfbed_M str5 Ix_selfbed_SD double Ix_rise_M str8 Ix_rise_SD double Ix_mid_M str5 Ix_mid_SD double Ix_DLMO_M str5 Ix_DLMO_SD
              "."     "."                          . ""                        . ""                           . ""                        . ""     
              ""      ""                           . ""                        . ""                           . ""                        . ""     
              ""      ""                           . ""                        . ""                           . ""                        . ""     
              ""      ""                           . ""                        . ""                           . ""                        . ""     
              "."     "."                          . ""                        . ""                           . ""                        . ""     
              ""      ""                           . ""                        . ""                           . ""                        . ""     
              ""      ""                           . ""                        . ""                           . ""                        . ""     
              ""      ""                           . ""                        . ""                           . ""                        . ""     
              "."     "."             -1893446820000 "01:25" -1893420360000.0002 "01:29"                      . ""           -1893446280000 "01:17"
              ""      ""         -1893450840000.0002 "01:28" -1893424619999.9998 "01:40"                      . ""           -1893441120000 "02:31"
              "11:17" "01:15:00"                   . ""           -1893433620000 "00:42"                      . ""                        . ""     
              "10:42" "00:49:00"                   . ""           -1.8934347e+12 "00:27"                      . ""                        . ""     
              "."     "."                          . ""                        . ""                           . ""                        . ""     
              ""      ""                           . ""                        . ""                           . ""                        . ""     
              "."     "."                          . ""                        . ""                           . ""                        . ""     
              ""      ""                           . ""                        . ""                           . ""                        . ""     
              ""      ""                           . ""                        . ""                           . ""                        . ""     
              "."     "."                          . ""                        . ""                           . ""                        . ""     
              ""      ""                           . ""                        . ""                           . ""                        . ""     
              ""      ""                           . ""                        . ""                           . ""                        . ""     
              "."     "."                          . ""           -1893426240000 " 2:10"                      . ""                        . ""     
              ""      ""                           . ""           -1893426780000 " 2:01"                      . ""                        . ""     
              "."     "."                          . ""                        . ""                           . ""                        . ""     
              ""      ""                           . ""                        . ""                           . ""                        . ""     
              ""      ""                           . ""                        . ""                           . ""                        . ""     
              "."     "."             -1893372360000 "00:34"      -1893430440000 "00:36"                      . ""           -1893442980000 "1.91" 
              ""      ""         -1893370619999.9998 "00:43"      -1893429360000 "00:51"                      . ""           -1893431760000 "1.71" 
              " 2:26" "01:14:00"                   . ""                        . ""                           . ""           -1893433440000 " 1:41"
              " 1:30" "01:17:59"                   . ""                        . ""                           . ""           -1893437160000 " 1:25"
              "."     "."                          . ""                        . ""                           . ""                        . ""     
              ""      ""                           . ""                        . ""                           . ""                        . ""     
              "."     "."                          . ""                        . ""                           . ""                        . ""     
              ""      ""                           . ""                        . ""                           . ""                        . ""     
              "."     "."                          . ""                        . ""                           . ""                        . ""     
              ""      ""                           . ""                        . ""                           . ""                        . ""     
              "."     "."                          . ""                        . ""                           . ""                        . ""     
              ""      ""                           . ""                        . ""                           . ""                        . ""     
              "."     "."                          . ""                        . ""                           . ""                        . ""     
              ""      ""                           . ""                        . ""                           . ""                        . ""     
              "11:09" "01:35:00"                   . ""      -1893432420000.0002 "12:49"                      . ""                        . ""     
              "6.1"   "22:19:12"                   . ""           -1892531520000 "28.93"                      . ""                        . ""     
              "10:17" "01:12:47"                   . ""      -1893434991818.1816 "01:51:31"                   . ""           -1893386580000 "12:13"
              "10:46" "01:10:46"                   . ""      -1893433557272.7273 "01:22:22"                   . ""           -1893383940000 "12:29"
              ""      ""                           . ""                        . ""                           . ""                        . ""     
              ""      ""                           . ""                        . ""                           . ""                        . ""     
              ""      ""                           . ""                        . ""                           . ""                        . ""     
              "."     "."                          . ""                        . ""                           . ""                        . ""     
              ""      ""                           . ""                        . ""                           . ""                        . ""     
              "12:58" "01:00:00"                   . ""      -1893425760000.0002 "12:53"                      . ""      -1893372839999.9998 " 1:00"
              "12:15" "00:53:59"                   . ""           -1.8934281e+12 "12:53"                      . ""           -1893377880000 "12:51"
              "12:18" "01:06:00"                   . ""      -1893427020000.0002 "12:57"                      . ""           -1893373680000 " 1:08"
              ""      ""                           . ""                        . ""                           . ""                        . ""     
              ""      ""                           . ""                        . ""                           . ""                        . ""     
              "."     "."                          . ""                        . ""                           . ""                        . ""     
              ""      ""                           . ""                        . ""                           . ""                        . ""     
              ""      ""                           . ""                        . ""                           . ""                        . ""     
              "."     "."                          . ""           -1.8934362e+12 "01:24"                      . ""      -1893448620000.0002 " 1:49"
              ""      ""                           . ""           -1893433260000 "01:09"                      . ""           -1893439980000 " 2:35"
              ""      ""                           . ""           -1893432780000 "00:52"                      . ""                        . ""     
              " 1:11" "01:30:00"                   . ""                        . ""                           . ""           -1.8933714e+12 ""     
              "12:38" "01:05:00"                   . ""                        . ""                           . ""           -1.8933804e+12 ""     
              " 1:41" "01:24:00"                   . ""                        . ""                           . ""           -1893453180000 "01:21"
              "12:55" "00:42:00"                   . ""                        . ""                           . ""           -1893414840000 "01:39"
              "12:40" "01:08:00"                   . ""                        . ""                           . ""           -1.8934152e+12 "01:35"
              "12:40" "00:15:00"                   . ""                        . ""                           . ""                        . ""     
              "11:07" "00:23:00"                   . ""                        . ""                           . ""                        . ""     
              "10:41" "00:08:00"                   . ""                        . ""                           . ""                        . ""     
              "."     "."             -1893372420000 " 1:00"      -1893426480000 " 1:01"                      . ""                        . ""     
              ""      ""              -1893372780000 " 1:04"      -1893428160000 "12:43"                      . ""                        . ""     
              ""      ""               -1.893372e+12 " 1:05"      -1893425940000 "12:56"                      . ""                        . ""     
              "."     "."                          . ""                        . ""                           . ""                        . ""     
              ""      ""                           . ""                        . ""                           . ""                        . ""     
              ""      ""                           . ""                        . ""                           . ""                        . ""     
              "."     "."                          . ""                        . ""                           . ""                        . ""     
              ""      ""                           . ""                        . ""                           . ""                        . ""     
              ""      ""                           . ""                        . ""                           . ""                        . ""     
              "."     "."                          . ""                        . ""                           . ""                        . ""     
              ""      ""                           . ""                        . ""                           . ""                        . ""     
              ""      ""                           . ""                        . ""                           . ""                        . ""     
              "."     "."                          . ""                        . ""                           . ""                        . ""     
              ""      ""                           . ""                        . ""                           . ""                        . ""     
              ""      ""                           . ""                        . ""                           . ""                        . ""     
              "."     "."                          . ""                        . ""                           . ""                        . ""     
              ""      ""                           . ""                        . ""                           . ""                        . ""     
              "."     "."                          . ""                        . ""                           . ""                        . ""     
              ""      ""                           . ""                        . ""                           . ""                        . ""     
              ""      ""                           . ""                        . ""                           . ""                        . ""     
              "."     "."                          . ""                        . ""                           . ""                        . ""     
              ""      ""                           . ""                        . ""                           . ""                        . ""     
              ""      ""                           . ""                        . ""                           . ""                        . ""     
              ""      ""                           . ""                        . ""                           . ""                        . ""     
              ""      ""                           . ""                        . ""                           . ""                        . ""     
              "."     "."                          . ""                        . ""                           . ""                        . ""     
              ""      ""                           . ""                        . ""                           . ""                        . ""     
              "10:37" "01:38:59"                   . ""      -1893436080000.0002 "00:58"                      . ""                        . ""     
              "10:54" "01:21:00"                   . ""           -1893435480000 "01:05"                      . ""                        . ""     
              "."     "."        -1893446400000.0002 "02:10"                   . ""              -1893425340000 "02:09"                   . ""     
              ""      ""              -1893450240000 "02:00"                   . ""              -1893432840000 "03:15"                   . ""     
              "."     "."             -1893448560000 "01:58"                   . ""              -1893427140000 "01:58"                   . ""     
              ""      ""              -1893450720000 "01:46"                   . ""         -1893436080000.0002 "02:06"                   . ""     
              end
              format %tchh:MM Ix_selfbed_M
              format %tcHH:MM:SS Ix_rise_M
              format %tchh:MM Ix_mid_M
              format %tchh:MM Ix_DLMO_M

              Comment


              • #8
                Yes having posted that I'm not confident the data exporting has worked. Sorry about this.

                Comment


                • #9
                  Some of those variables read in as intended and some didn't. I see the name excel (NB: should be Excel) and fear the worst.

                  With the string variables that were not read in as numeric date-times the culprit is likely to be one or more of

                  1. Header material or other metadata that should have been excluded.

                  2. Symbols that are not used by Stata. For example, NA may look obvious to a person but has no special meaning to Stata.

                  3. Extra gunk, e.g. results that someone added below and beyond the data in a worksheet. I often see cases where students add extra calculations in a corner and forget to tell me about them before import into Stata.

                  Ix_SOT_M Ix_SOT_SD Ix_selfbed~D Ix_rise_SD Ix_mid_SD Ix_DLMO_SD are string variables and probably shouldn't be. I would loop over all of those and check for values that inhibited reading as date-times.

                  In fact, look at those variables to see some real problems:

                  Code:
                  . ds, has(type string)
                  Ix_SOT_M      Ix_SOT_SD     Ix_selfbed~D  Ix_rise_SD    Ix_mid_SD     Ix_DLMO_SD
                  
                  . tab1 `r(varlist)'
                  
                  -> tabulation of Ix_SOT_M  
                  
                     Ix_SOT_M |      Freq.     Percent        Cum.
                  ------------+-----------------------------------
                         1:11 |          1        2.04        2.04
                         1:30 |          1        2.04        4.08
                         1:41 |          1        2.04        6.12
                         2:26 |          1        2.04        8.16
                            . |         28       57.14       65.31
                        10:17 |          1        2.04       67.35
                        10:37 |          1        2.04       69.39
                        10:41 |          1        2.04       71.43
                        10:42 |          1        2.04       73.47
                        10:46 |          1        2.04       75.51
                        10:54 |          1        2.04       77.55
                        11:07 |          1        2.04       79.59
                        11:09 |          1        2.04       81.63
                        11:17 |          1        2.04       83.67
                        12:15 |          1        2.04       85.71
                        12:18 |          1        2.04       87.76
                        12:38 |          1        2.04       89.80
                        12:40 |          2        4.08       93.88
                        12:55 |          1        2.04       95.92
                        12:58 |          1        2.04       97.96
                          6.1 |          1        2.04      100.00
                  ------------+-----------------------------------
                        Total |         49      100.00
                  
                  -> tabulation of Ix_SOT_SD  
                  
                    Ix_SOT_SD |      Freq.     Percent        Cum.
                  ------------+-----------------------------------
                            . |         28       57.14       57.14
                     00:08:00 |          1        2.04       59.18
                     00:15:00 |          1        2.04       61.22
                     00:23:00 |          1        2.04       63.27
                     00:42:00 |          1        2.04       65.31
                     00:49:00 |          1        2.04       67.35
                     00:53:59 |          1        2.04       69.39
                     01:00:00 |          1        2.04       71.43
                     01:05:00 |          1        2.04       73.47
                     01:06:00 |          1        2.04       75.51
                     01:08:00 |          1        2.04       77.55
                     01:10:46 |          1        2.04       79.59
                     01:12:47 |          1        2.04       81.63
                     01:14:00 |          1        2.04       83.67
                     01:15:00 |          1        2.04       85.71
                     01:17:59 |          1        2.04       87.76
                     01:21:00 |          1        2.04       89.80
                     01:24:00 |          1        2.04       91.84
                     01:30:00 |          1        2.04       93.88
                     01:35:00 |          1        2.04       95.92
                     01:38:59 |          1        2.04       97.96
                     22:19:12 |          1        2.04      100.00
                  ------------+-----------------------------------
                        Total |         49      100.00
                  
                  -> tabulation of Ix_selfbed_SD  
                  
                  Ix_selfbed_ |
                           SD |      Freq.     Percent        Cum.
                  ------------+-----------------------------------
                         1:00 |          1        9.09        9.09
                         1:04 |          1        9.09       18.18
                         1:05 |          1        9.09       27.27
                        00:34 |          1        9.09       36.36
                        00:43 |          1        9.09       45.45
                        01:25 |          1        9.09       54.55
                        01:28 |          1        9.09       63.64
                        01:46 |          1        9.09       72.73
                        01:58 |          1        9.09       81.82
                        02:00 |          1        9.09       90.91
                        02:10 |          1        9.09      100.00
                  ------------+-----------------------------------
                        Total |         11      100.00
                  
                  -> tabulation of Ix_rise_SD  
                  
                   Ix_rise_SD |      Freq.     Percent        Cum.
                  ------------+-----------------------------------
                         1:01 |          1        4.35        4.35
                         2:01 |          1        4.35        8.70
                         2:10 |          1        4.35       13.04
                        00:27 |          1        4.35       17.39
                        00:36 |          1        4.35       21.74
                        00:42 |          1        4.35       26.09
                        00:51 |          1        4.35       30.43
                        00:52 |          1        4.35       34.78
                        00:58 |          1        4.35       39.13
                        01:05 |          1        4.35       43.48
                        01:09 |          1        4.35       47.83
                     01:22:22 |          1        4.35       52.17
                        01:24 |          1        4.35       56.52
                        01:29 |          1        4.35       60.87
                        01:40 |          1        4.35       65.22
                     01:51:31 |          1        4.35       69.57
                        12:43 |          1        4.35       73.91
                        12:49 |          1        4.35       78.26
                        12:53 |          2        8.70       86.96
                        12:56 |          1        4.35       91.30
                        12:57 |          1        4.35       95.65
                        28.93 |          1        4.35      100.00
                  ------------+-----------------------------------
                        Total |         23      100.00
                  
                  -> tabulation of Ix_mid_SD  
                  
                    Ix_mid_SD |      Freq.     Percent        Cum.
                  ------------+-----------------------------------
                        01:58 |          1       25.00       25.00
                        02:06 |          1       25.00       50.00
                        02:09 |          1       25.00       75.00
                        03:15 |          1       25.00      100.00
                  ------------+-----------------------------------
                        Total |          4      100.00
                  
                  -> tabulation of Ix_DLMO_SD  
                  
                   Ix_DLMO_SD |      Freq.     Percent        Cum.
                  ------------+-----------------------------------
                         1:00 |          1        6.25        6.25
                         1:08 |          1        6.25       12.50
                         1:25 |          1        6.25       18.75
                         1:41 |          1        6.25       25.00
                         1:49 |          1        6.25       31.25
                         2:35 |          1        6.25       37.50
                        01:17 |          1        6.25       43.75
                        01:21 |          1        6.25       50.00
                        01:35 |          1        6.25       56.25
                        01:39 |          1        6.25       62.50
                        02:31 |          1        6.25       68.75
                         1.71 |          1        6.25       75.00
                         1.91 |          1        6.25       81.25
                        12:13 |          1        6.25       87.50
                        12:29 |          1        6.25       93.75
                        12:51 |          1        6.25      100.00
                  ------------+-----------------------------------
                        Total |         16      100.00

                  Comment


                  • #10
                    I thought I would update here for the benefit of anyone who is subsequently searching for answers to a similar question - the answer to my problem was that I didn't need to use time variables after all, they were presented as times in the studies so I had extracted them as such in my data extraction table, but as I needed to use them as mean values for outcomes post intervention, not to plot a timeseries or anything like that, I needed to turn them into numeric variables.

                    There may have been a clever way to do this in Stata, but I did it reasonably quickly in Excel, turning into either just minutes, or hours and decimals of hours. If you have time points either side of midnight (e.g. sleep onset time) watch out that you probably want 1am to be later than 11:30pm, so I made a formula that added 24 to all values under 10, which I used on the variables presenting times around nighttime/early hours. Hope someone finds that useful!

                    Comment

                    Working...
                    X