Which AI is most efficient for helping me with Python?

July 1, 2023 10:19 PM

I am not a coder but sometimes use Python (with numpy etc) for generating “in silico” datasets I need or running calculations on such datasets and have found ChatGPT to be a really useful tool for helping with that. That said….

ChatGPT is never right on the first try - it inevitably gives me code that then generates errors. After a few iterations where I basically tell ChatGPT the error code I’m seeing, I generally get what I need on most requests but some have just been total fails, where ChatGPT cannot provide code that doesn’t error. My knowledge of AI is pretty skimpy though, and I find myself wondering if moving to something besides ChatGPT could work better.

Also I’m using python because that’s what ChatGPT told me to do when I simply made the request for the datasets (ChatGPT replied with, here’s a python script for that, but I am just a text based system and can’t execute the script for you.). Is Python the best thing for this? Should I ask ChatGPT a different kind of code?

Example of the kinds of datasets I need generated (and how I phrase it when I ask ChatGPT for help): a set of 20 groups of integers, where each group comprises 4 integers, for a total of 80 integers, where the mean of all 80 integers is 100, the standard deviation of each groups is 5 or less and all values in the set fall between 50 and 150.

Or, repeat the generation above 100 times and report the maximum and minimum standard deviation observed on the sets.

The first example is of the type where ChatGPT usually gets there quickly but requests of the second type can take a while or just fail with errors that I don’t understand.

posted by Tandem Affinity to Technology (12 answers total)

Most efficient or most correct? As amazing as LLM's are, do not trust any of these. They hallucinate. That's the current term for built in inaccuracy that usually seems right. ChatGPT-4 seems to be the top of the line at the moment.
posted by sammyo at 10:38 PM on July 1, 2023 [3 favorites]

Have you tried GitHub Copilot? Unlike ChatGPT, it’s specifically designed for generating code.

That said, I don’t think there’s currently ANY model publicly available that’s good at responding to the type of request in your post. Generative models just aren’t good at specifics and mathematical precision (yet). Though I imagine we might get there within a few years.
posted by mekily at 10:56 PM on July 1, 2023 [1 favorite]

GPT 4 is far better. Maybe try a different technique - describe the output you want, then tell it to ask you question until it's clear. Then ask it to discuss the best way to implement the code, then ask for code. Much better results.
posted by lookoutbelow at 11:03 PM on July 1, 2023 [1 favorite]

Best answer: Is Python the best thing for this? Should I ask ChatGPT a different kind of code?

python is a perfectly good thing for this, R is also a very good option.
If you're getting ChatGPT to generate code for you, it has probably consumed more "how to numpy" blogs/githubs than "how to R" blogs/githubs. The quality of those texts presumably varies - NFI which makes ChatGPT give more useful responses though.

Refining your prompt is crucial. Don't expect magic on the first ask.
posted by pompomtom at 4:08 AM on July 2, 2023

Consider learning how to generate this stuff with Python (or R, or Octave) using formal code and standard books and tutorials, written by humans who understand this stuff and who have some idea how to teach.

I really think that will be easier and produce better results than learning how to coax a mostly black-box chat bot to produce formal code using natural language and then bug hunting the outputted code and then checking the the output of that code manually, because you can't trust a this chat bot (at all!) ,and then hoping that you understand some pieces of this process enough to not miss huge mistakes.

You don't say what your application is, but if it's anything you care about and want to do right, just learn the appropriate tool, rather than fumble with a currently trending inappropriate tool.
posted by SaltySalticid at 5:14 AM on July 2, 2023 [9 favorites]

Best answer: Your general approach is fine. Python is a perfectly good language for producing this sort of output. Generating simple Python code is one of the things that ChatGPT does well. I recommend ChatGPT 4 rather than 3 - the output will be much better. ChatGPT certainly can hallucinate, but in my experience it doesn't often for a task like this, and since you can immediately execute the code to check its output then it doesn't really matter if it does.

However, I recommend more rigour in your prompts. Make them clearer, and when you have examples of what you want, include them in the prompt.

I reformulated your prompt in the question to be:

Please write some code in Python.

This code should not use any external libraries beyond the Python standard library.

When run, it should output a set of 20 groups of integers, matching these criteria. Each group comprises 4 integers between 50 and 150, for a total of 80 integers. The mean of all 80 integers must be 100. The standard deviation of each group must be 5 or less.

It should also output the mean of all the integers. It should also output the standard deviation of each group.

Please include tests for each of the criteria that assert that the output is correct.


and I got this response - https://pastebin.com/s52137U4 - which is valid Python and matches my understanding of your requirements. If you want a mean of precisely 100, rather than close to 100 (when I ran it, I got 100.1), then you can refine your prompt accordingly. (I limited it to the standard library just because it made it easier for me to run via Jupyter - you don't need to assuming you are happy to install libraries)
posted by siskin at 5:47 AM on July 2, 2023 [3 favorites]

Because ChatGPT 4 has a limit of 25 messages every 3 hours, I sometimes use ChatGPT 3.5 to refine my prompts before using them in 4 to produce final code.
posted by donpardo at 6:48 AM on July 2, 2023 [1 favorite]

Best answer: I'm inclined to think that needing to ask this question means you're reaching the point where an hour or two with the numpy documentation and StackOverflow would be a better investment and save you time in the long run. I know you say you're "not a coder" but you've gotten as far as you have, so you've presumably surmounted the "install numpy" hurdle (rereading your question, you don't mention numpy, but I think that's what you have to be using) and the "make it run" hurdle. Those are honestly harder hurdles to get over as a beginner than the "I have a script that does X and I need to make it do Y" hurdle.
posted by hoyland at 6:59 AM on July 2, 2023 [5 favorites]

Best answer: The answer to your question at time of writing is probably that Github Copilot X is the most efficient AI for helping with basic code generation. I've been part of a private beta for a couple of months, and it is much, much better than regular ChatGPT at generating functional code and fixing bugs.

It doesn't have some of the fancier code review features that CodeWhisperer and Bard offer, but it sounds like you don't need those anyway.

That said, there are a few caveats:

1) Copilot X is meant is meant to integrate with an IDE like Visual Studio Code, so if you aren't using one of those you're out of luck.

2) It's a private beta for paid subscribers, which means you have to wait a few weeks to get an invite and pay about $10 / month for GitHub's existing Copilot service.

3) It still makes mistakes, so you have to understand enough about Python (or whatever language and/or libraries you are using) to be able to tell where the problems are and how to troubleshoot them when your code's not working properly.

That said, Copilot X is really good at answering questions about how code works and giving examples. So I've found it extremely useful as a learning tool -- as long as you approach its answers with the appropriate level of skepticism and critical thought.
posted by mcbaya at 7:49 AM on July 2, 2023 [2 favorites]

If you can get it, I recommend using Wolfram Mathematica. The latest version (13.3) has integrated ChatGPT into the notebook itself, so it's a lot simpler. I just tried out your prompt and it seems to work reasonably well.
posted by dhruva at 8:45 AM on July 2, 2023 [1 favorite]

Response by poster: OMG I feel so validated because getting numpy going really was the hardest part about this whole process. I hear you all about just learning to write the code myself and I think that will happen organically as I see more and more examples of how it works, and then I can get more formal instruction to take it all the way. (For those worried about the AI hallucinations, which I totally get, I felt the AI route and any possible risk in poor results was acceptable, as I'm generating these as dummy datasets to illustrate how some statistical concepts play out in our process to others, so I would be able to see if I got really wrong results. I'm basically using it as a random number generator, but with a lot of constraints on "random". Maybe if I can get a better handle on this without needing the AI, I will start doing more 'important' things that can afford less risk.)
posted by Tandem Affinity at 6:29 PM on July 2, 2023 [1 favorite]

Probably NOT what you're asking for, but I would take a course on Coursera on the subject of Data Science. You can probably audit the course and not pay a penny (you just don't get the certificate to share with LinkedIn or your employer) I took the IBM version, but Google has one too. That teaches you a bunch about numpy, scikit, and all that jazz.
posted by kschang at 5:15 AM on July 3, 2023