Using required error

[Sumedha Gupta]

Dear All,
I want to download a .csv file from a website and process it. The files are available as separately for each month and each variable. So I am trying to write a short loop but get an error:

. clear

. /*import delimited "$download_site_file_path/2020/05/prcp-202005-ste-prelim.csv"
>         drop v1 v2
>         rename (v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 v13 v14 v15 v16 v17 v18 v19 v20 v21 v22 v23 v24 v25
>  v26 v27 v28 v29 v30 v31 v32 v33 v34 v35 v36 v37) ///
>         (Statename Year Month cli_measure day1 day2 day3 day4 day5 day6 day7 day8 day9 day10 day11 d
> ay12 day13 day14 day15 day16 day17 day18 day19 ///
>         day20 day21 day22 day23 day24 day25 day26 day27 day28 day29 day30 day31)
> drop cli_measure
> reshape long day, i(Statename) j(Day)
> rename day prcp
> recode prcp -999.98999=.
>         */
. forval M = 1/1{
  2.     foreach var in prcp {
  3.         import delimited "$download_site_file_path/2020/0"`M'"/"`var'"-20200"`M'"-ste-prelim.csv"
  4.         drop v1 v2
  5.         rename (v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 v13 v14 v15 v16 v17 v18 v19 v20 v21 v22 v23 v24 
> v25 v26 v27 v28 v29 v30 v31 v32 v33 v34 v35 v36 v37) ///
>         (Statename Year Month cli_measure day1 day2 day3 day4 day5 day6 day7 day8 day9 day10 day11 d
> ay12 day13 day14 day15 day16 day17 day18 day19 ///
>         day20 day21 day22 day23 day24 day25 day26 day27 day28 day29 day30 day31)
  6.         drop cli_measure
  7. reshape long day, i(Statename) j(Day)
  8. rename day prcp
  9. recode prcp -999.98999=.
 10.         save "$rawdata_file_path\`var'_2020`M'.dta", replace
 11. }
 12. 
. }
using required
r(100);

end of do-file

r(100);

What am I doing wrong?

Many thanks in advance for your help.
Sincerely,
Sumedha.

[Stephen Jenkins]

It's hard to tell given what you're showing us! I would suggest doing some experiments to check that the macros in the expression

 
 "$download_site_file_path/2020/0"`M'"/"`var'"-20200"`M'"-ste-prelim.csv"

are being expanded as you intend. Perhaps you need also to check out "compound double quotes".

[Sumedha Gupta]

Thank you for your help. How may I check that its expanded correctly?

[William Lisowski]

I believe your use of quotation marks in file paths is confusing Stata, causing it to issue a confusing error message. Try replacing

import delimited "$download_site_file_path/2020/0"`M'"/"`var'"-20200"`M'"-ste-prelim.csv"

with

import delimited "$download_site_file_path/2020/0`M'/`var'-20200`M'-ste-prelim.csv"

(removing all the double-quote marks between the outermost pair) and replacing

save "$rawdata_file_path\`var'_2020`M'.dta", replace

with

save "$rawdata_file_path/`var'_2020`M'.dta", replace

(replacing the back-slash with a forward-slash).

To address your question in post #3,

. global download_site_file_path /a/b

. local var prcp

. local M 1

. local ex1 "$download_site_file_path/2020/0"`M'"/"`var'"-20200"`M'"-ste-prelim.csv"

. local ex2 "$download_site_file_path/2020/0`M'/`var'-20200`M'-ste-prelim.csv"

. macro list _ex1 _ex2
_ex1:           /a/b/2020/0"1"/"prcp"-20200"1"-ste-prelim.csv
_ex2:           /a/b/2020/01/prcp-202001-ste-prelim.csv

[Sumedha Gupta]

Prof. Lisowski,
Thank you so much. This is super helpful. Unfortunately it does not solve my problem completely yet. So, using your code i figures that _ex2 in #4 is how i should write it. I have also tried to simplify it by removing one of the loops. But still, I get an error.

.         
.         forval M=1/5{
  2.             clear
  3.         import delimited "$download_site_file_path/2020/0`M'/prcp-20200`M'-ste-prelim.csv"
  4.         drop v1 v2
  5.         rename (v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 v13 v14 v15 v16 v17 v18 v19 v20 v21 v22 v23 v24 
> v25 v26 v27 v28 v29 v30 v31 v32 v33 v34 v35 v36 v37) ///
>         (Statename Year Month cli_measure day1 day2 day3 day4 day5 day6 day7 day8 day9 day10 day11 d
> ay12 day13 day14 day15 day16 day17 day18 day19 ///
>         day20 day21 day22 day23 day24 day25 day26 day27 day28 day29 day30 day31)
  6.         drop cli_measure
  7. reshape long day, i(Statename) j(Day)
  8. rename day prcp
  9. recode prcp -999.98999=.
 10.         save "$rawdata_file_path/prcp_2020`M'.dta", replace
 11. }
could not open url
r(603);

end of do-file

r(603);

However, it works perfectly when I take out the loop:

 clear

. import delimited "$download_site_file_path/2020/05/prcp-202005-ste-prelim.csv"
(37 vars, 48 obs)

Not sure why the loop messes it up. I will be grateful for any help you may be able to offer.
Sincerely,
Sumedha.

[William Lisowski]

Your message is different now: it tells you Stata could not open the URL I will note that it failed for M==1 but you show us a successful command with M==5 so your comparison is unconvincing.

Since you haven't shared the URL with us there's not much more I can tell you; I can't even try to replicate your problem. Please post in a code block the following commands and their output.

local M 1
local path "$download_site_file_path/2020/0`M'/prcp-20200`M'-ste-prelim.csv"
macro list download_site_file_path _path
import delimited "$download_site_file_path/2020/0`M'/prcp-20200`M'-ste-prelim.csv"

[Sumedha Gupta]

Prof. Lisowski,
Thank you so much for your help. You pointed out that in my working example M=5 whereas in my non-working example M=1. I checked and for some reason the file name on the web resource varied slightly between M=5 and M=1, causing the error. I greatly appreciate your help.
Sincerely,
Sumedha.