Looking for an option to modify the COPY command

[Artur Deguzman]

Hello all,

I am using the following expression to download data from a portal (backend Microsoft SQL):

clear
copy https://data.sccgov.org/resource/s3fb-yrjp.csv data_from_socrata2.csv, replace

It only gets me 1000 observations.

I talked to an IS guy and said that I could increase the 1000 limit, but he does not know Stata.

I looked at the COPY command documentation and could not find any options for increasing the limit.

In parallel with the above method, I am exploring using JSON downloading (or bringing into STATA) options. If you have any solutions for them, I am interested in them as well.

Does anyone have a solution?

Thanks!

[Leonardo Guizzetti]

Copy is basically a utility command for the system copy command. I just used your link and my downloaded file only contains 1,000 observations. The rate/size limitation is being set by the server, so you'll need to figure out how to get more data from the governmental organization.

[Artur Deguzman]

Copy is basically a utility command for the system copy command. I just used your link and my downloaded file only contains 1,000 observations. The rate/size limitation is being set by the server, so you'll need to figure out how to get more data from the governmental organization.

Hello Leonardo,

I found the solution but there is still a problem when I put the expression in a DO file:

• Via a browser, this expression enables me to get more than 1000 lines: https://data.sccgov.org/resource/s3fb-yrjp.csv?$limit=50000
• When I plug it into my DO file, I get the following error message: =exp not allowed.

I encoded the "=" as a %40, but it did not work. There needs to be an escape code.

What would that escape code be that is acceptable to Stata?

Thanks!

[Leonardo Guizzetti]

The "$" is a flag to Stata's interpreter than what follows is a global macro. To escape this, put a single backslash before the "$".

copy "https://data.sccgov.org/resource/s3fb-yrjp.csv?\$limit=50000" data_from_socrata2.csv, replace

I did this and now I get back about 25,000 rows.

[Artur Deguzman]

The "$" is a flag to Stata's interpreter than what follows is a global macro. To escape this, put a single backslash before the "$".

copy "https://data.sccgov.org/resource/s3fb-yrjp.csv?\$limit=50000" data_from_socrata2.csv, replace

I did this and now I get back about 25,000 rows.

Thank you!

I love this forum!