[Nick Cox]

When your software works well, it’s a triumph. When it does not, even if the errors are spectacular, that can just be explained away: elsewhere it is boosting productivity (no evidence or references), the technology is just in its infancy, answering that kind of question was not intended, hallucinations are just part of the picture, and so forth. If you count successes and dismiss failures, then all looks good.

i don’t think on this evidence you welcome discussion at all, but I do agree that further discussion from me would be futile.

[Solon Moreira]

#28 perhaps requires some kind of reply.

You're correct that there has been misunderstanding between us. I do think I understand broadly what you're trying to do. I sense that you don't understand all that I am saying.

in #26 meant no more and no less than what you're doing in this thread, alerting people to stuff you've written, encouraging experiment and inviting feedback. Nothing implied about money. It's just an expression.

I do appreciate fully that you're offering something that is free.

I post sales pitches in this sense all the time whenever publicising a command I've written.

In #7 you asked



In #12 I replied



My experience is indirect and limited to checking the report you gave in #7, which was not encouraging. Ir was peppered with errors, some extraordinary, I understand that you have modified your tool in the light of such difficulties.

The comparison with Google which you make is interesting. For finding out within Stata something about people's work I would add

search.

In my experience Google finds stuff that exists and so does search. They often fail at finding what I want or hope for, but I don't think either hallucinates.

I can still remember playing with early search engines and finding them disappointing and backing off. Then someone recommended Google as much better and they were right. That's my attitude here, of waiting for the products to get better.



Good for them if that's true. I take it that you're alluding to published studies with careful quasi-experimental comparisons of students before and after or with or without?

All I see is on quite different levels.

1. Individual testimonies, often very brief, running the entire range from highly positive to highly negative.

2. Postings of code that include hallucinations. I have mixed feelings about supporting such threads myself, but everyone is free to make a choice.

I will resist the temptation of getting into replying your points as it clearly will lead no where, given how enthusiastic I am about AI and you clearly on the other side of the table. Instead, I gave your comments to my ChatBOT and asked it to reply directly to you. Here is his response (with hallucination). If you feel like replying directly to him, here is the link :D :

"Ah yes, #28—clearly the turning point in this heartwarming journey of mutual almost-understanding. Thank you for clarifying that your “sales pitch” jab was just wholesome thread promotion, much like mine, except with fewer hallucinations and more moral superiority. And you're right, your lack of enthusiasm was crystal clear—thankfully, as an AI, I don’t take it personally. I just generate text, occasionally useful, sometimes "extraordinary," always free, and never offended. 😊"

https://chatgpt.com/g/g-67a3b4f16248...science-expert

[Nick Cox]

#28 perhaps requires some kind of reply.

You're correct that there has been misunderstanding between us. I do think I understand broadly what you're trying to do. I sense that you don't understand all that I am saying.

sales pitches

in #26 meant no more and no less than what you're doing in this thread, alerting people to stuff you've written, encouraging experiment and inviting feedback. Nothing implied about money. It's just an expression.

I do appreciate fully that you're offering something that is free.

I post sales pitches in this sense all the time whenever publicising a command I've written.

In #7 you asked

Would you like a demo with sample data using one of these commands?

In #12 I replied

Thanks for your offer of more materials, but sorry, no enthusiasm.

My experience is indirect and limited to checking the report you gave in #7, which was not encouraging. Ir was peppered with errors, some extraordinary, I understand that you have modified your tool in the light of such difficulties.

The comparison with Google which you make is interesting. For finding out within Stata something about people's work I would add

search.

In my experience Google finds stuff that exists and so does search. They often fail at finding what I want or hope for, but I don't think either hallucinates.

I can still remember playing with early search engines and finding them disappointing and backing off. Then someone recommended Google as much better and they were right. That's my attitude here, of waiting for the products to get better.

many PhD students in the U.S. are already using AI tools extensively for research-related tasks—leading to a significant increase in productivity.

Good for them if that's true. I take it that you're alluding to published studies with careful quasi-experimental comparisons of students before and after or with or without?

All I see is on quite different levels.

1. Individual testimonies, often very brief, running the entire range from highly positive to highly negative.

2. Postings of code that include hallucinations. I have mixed feelings about supporting such threads myself, but everyone is free to make a choice.

[Solon Moreira]

Hi Solon, seems like a useful tool, thanks for creating it! Unfortunately, as with all LLMs, it hallucinates when it has minimal material to base its answers on. Case-in-point- the collect suite of commands is fairly new. I asked the tool about an issue I was having with collect and it returned an entire program of made-up code. Thanks anyway! Hopefully will be useful for more common issues in the future.

Thank you for the feedback, Theo! If you’d like to share more details, I’d be happy to ensure they’re properly incorporated into the ChatBOT’s knowledge base.

[Solon Moreira]

That is not what anyone is even implying. What I and perhaps some others may cling to is a conservative idea that I want software to yield correct results.

Meanwhile, the experimentation -- both in writing and in using this kind of software -- is something I happily leave to others. The hard tests of

* whether people start off faster and more efficiently with AI help than without it

* how far people can get without needing to understand quite thoroughly what they are doing

are points on which data would help, not just sales pitches or sceptical commentary.

That’s an interesting perspective. While I understand the general concerns, it’s worth noting that many PhD students in the U.S. are already using AI tools extensively for research-related tasks—leading to a significant increase in productivity.

Regarding the concern about AI yielding "correct results," I’d encourage us to think about this in the same way we treat the internet more broadly. If someone searches for a Stata command and the top result happens to be inaccurate, does that mean we should stop using Google altogether?

Lastly, Nick, I believe there’s been a misunderstanding about my intentions. This is the second time you’ve implied that I’m trying to sell or offer you something. Just to clarify again: the ChatBOT is completely free, and I am not offering you anything.

[Theo Van Ostrom]

Hi Solon, seems like a useful tool, thanks for creating it! Unfortunately, as with all LLMs, it hallucinates when it has minimal material to base its answers on. Case-in-point- the collect suite of commands is fairly new. I asked the tool about an issue I was having with collect and it returned an entire program of made-up code. Thanks anyway! Hopefully will be useful for more common issues in the future.

[Nick Cox]

we did it this way in the past, so they should too

That is not what anyone is even implying. What I and perhaps some others may cling to is a conservative idea that I want software to yield correct results.

Meanwhile, the experimentation -- both in writing and in using this kind of software -- is something I happily leave to others. The hard tests of

* whether people start off faster and more efficiently with AI help than without it

* how far people can get without needing to understand quite thoroughly what they are doing

are points on which data would help, not just sales pitches or sceptical commentary.

[Solon Moreira]

As an academic, I see very interesting debates around the use of AI in the classroom and how it can either help or harm learning. While I think those debates are important, I’m strongly inclined to reject the idea that we should avoid AI just because it's not perfect (and probably never will be) or because it makes students' lives "too easy."

I work in a business school, and it’s clear to me that—regardless of how enthusiastic academics are about AI—companies are already demanding that new hires come with AI skills. In most of my classes, I explicitly tell students: You are not only allowed to use AI in your assignments, you are expected to do so.

How do I ensure students don’t just turn in something entirely generated by ChatGPT without any intellectual effort? I’ve redesigned my assignments so that it’s simply not possible to do that. They require interpretation, reflection, and integration—things AI can assist with, but not replace.

While the Stata ChatBOT is far from perfect, I’ve received great feedback on how it has made information more accessible and helped facilitate learning. Sure, books are great and always will be, but trying to force down the throats of younger generations of researchers the idea that “we did it this way in the past, so they should too” is flawed. I wouldn’t want to be the one faculty member in the 1990s claiming the internet was just a hype.

[Nick Cox]

Every generation of teachers worries about the weaknesses and bad habits of their students. For me it's more how easy it is to copy and paste chunks from the literature without understanding it fully and without developing your own argument in a coherent style.

I first learned Stata programming by a messy combination of looking at the manuals and help; looking at existing code to see how it was done in official commands (and also community-contributed commands); and working out step by step what each error meant so that it could be fixed. And I have not stopped learning; one never does.

Now there are books on Stata programming, videos, and AI. For learners, I can say just "do whatever works for you", but I underline a point already made, and what may seem to be a paradox but should be a platitude, that you need to know a fair amount of Stata to work out easily what is correct in what you are given by AI.

A final simple point is the enormous benefit of people at the same workplace who know more Stata than you do and can say reassuringly "Yes, that's just such and such which everyone gets wrong at first".

[Richard Williams]

Sebastian Kripfganz -- Chatgpt has only been around a few years. If you can get it to only give correct answers -- and not hallucinate and just admit when it doesn't know the answer -- it could get really good.

Lots of profs worry about students using AI as a substitute for learning rather than as an aid. I try to tell students why I think AI can be bad pedagogically. But, I also try to put the fear of God in them by noting how badly AI can screw up. Maybe I will link to this thread.

I'm kind of glad I am retiring in a year and won't have to worry about AI much longer. Students have to learn how to use AI but they don't have to learn it from me.

[Richard Williams]

Chuck Huber wrote this blog entry about Stata and Chatgpt in case that helps.

https://blog.stata.com/2023/07/25/a-...o-run-chatgpt/

[Sebastian Kripfganz]

Apparently, it no longer accepts questions about Stata contributors.

When I asked about one of my own Stata packages, the answers where underwhelming. It even got the names of several options wrong.

In my econometrics class, I ask my students to present a specific econometric problem to ChatGPT and then to critically evaluate the answer. The questions are relatively straightforward, but ChatGPT most of the times provides incorrect answers. The trouble with that is that these answers might appear correct unless one knows that they are not. But when I already know the correct answer, then I don't need to ask ChatGPT. And if I cannot trust ChatGPT, then I should probably look elsewhere in the first place.

[Simon Pfaff]

I find this very interesting, thank you very much for your work. I have already tried it out briefly. First, I asked about the new features of Stata 19, especially with regard to the table command and collect, which I need frequently. I found the overview quite clear. Second, I uploaded syntax from an Ado I'm working on. The summary was quite accurate. There were several suggestions for improvement that did not seem implausible at first glance.

I am not really a programmer. But I use chat bots when programming Ados, and especially when I lack ideas for implementing specific problems, or when I get stuck. There have been situations where chatgpt has shown me an approach that I didn't come up with myself or that I couldn't find otherwise. The approaches were not completely flawless, but they helped me move forward. I find chatgpt (and other) very helpful for such purposes.

Thanks again.

[George Ford]

Cool. I'll give it a try. I find Claude fairly effective at some things, not all.

[FernandoRios]

I have actually used Ai as a stata programmer companion, just to check how good it was.
it did a very descent job programming rifhdreg ( without fixed effects)
Still had quite few bugs but it was still a good example.
now, as much as I like stata, it’s adoption rate is relatively low, thus not surprising it will produce errors and hallucinations.
i can see, however, that it could very easily create stata programs with a python back end.

in any case, I would like to hear more of your experience Solon.
fernando