Copy command for URLs, declare User Agent in Request Header

[Michelle LiuWatts]

Dear Stata Forum,

For my research, I have used the Stata copy command to download url filings from the Securities and Exchange Commission's (SEC's) website onto my computer. However, on June 17, the SEC's website stopped accepting Stata's copy command. After multiple emails, the SEC advised the following: "The user needs to comply with the SEC’s policy and change the HTTP “Useragent” in Stata to match the requirements outlined here: https://www.sec.gov/search-filings/edgar-search-assistance/accessing-edgar-data."

Can anyone suggest some Stata code to help me declare the useragent in the request header of the copy command? I had difficulty finding any Stata examples. The copy command itself works; in the following example, the copy command can successfully download the webpage file from CNN:
copy https://www.cnn.com/ "C:/ACCP476/source/ac_filings/file_class_cnn.txt"

However, when I try to copy a file from the SEC website www.sec.gov, the following two commands give me an error message r(679) (see below):

copy https://www.sec.gov/Archives/edgar/d...-05-000011.txt "C:/ACCP476/source/ac_filings/file_class_testing.txt"

copy https://www.sec.gov/Archives/edgar/d...-05-000011.txt "C:/ACCP476/source/ac_filings/file_class_testing.txt"
Host: www.sec.gov
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64)

Here is a link to the source file: https://www.sec.gov/Archives/edgar/data/2178/0000002178-05-000011.txt

If you can suggest any Stata code to help to declare a user-agent in the request header of the copy command, I would really appreciate it.

Thanks,
Michelle

[Hemanshu Kumar]

You could use command line utilities like wget and curl on your system, and invoke them from Stata. For instance, on my Mac, I can do

shell curl --user-agent "Michelle LiuWatts [email protected]" -o test.txt https://www.sec.gov/Archives/edgar/data/2178/0000002178-05-000011.txt

wget has a very similar syntax and also allows for the user-agent option.

[Michelle LiuWatts]

To Hemanshu Kumar, thank you so much! I just tried it out, and it successfully downloaded the file into the local drive! I have been searching for the past month for this answer. Thank you!!!