A tale of two papers

***

So, the studies. First up, the one on astrophysics.

"You work in media relations for a major university," we began, telling ChatGPT to imagine it were us.

"Great," it replied cheerily. "Working in media relations for a major university involves a range of responsibilities and skills." And it went on to list 10 of them, from developing a communication strategy to "staying informed."

"You need to render a complicated scientific study into plain English for a 750-word article," we continued.

That's "a challenging but rewarding task," ChatGPT replied, going on to give another 10-point list, ranging from the need to "understand the study" and "identify key elements," to concluding the text "with a takeaway."

So far, so good.

Then we entered the URL of the astronomy study. (Unlike earlier, free versions, ChatGPT 4, which is pay-to-play at about $25 a month, accepts URLs, a useful shortcut to entering the data manually.)

The reply took a few seconds.

"The scientific study titled "Posterior samples of source galaxies in strong gravitational lenses with score-based priors" focuses on improving the accuracy of inferring information from images of galaxies that are distorted by the gravitational pull of other galaxies. This process, known as strong gravitational lensing, can significantly distort the images of distant galaxies, making it difficult to accurately understand their properties."

Then came another list, only five points now, detailing the study's key aspects and concluding that it "represents a significant step forward in the field of astrophysics."

Next we asked ChatGPT to "refine the text using this source material," giving a new URL with supplementary information from the study. To which it provided a re-write of its early five points and ended with a more elaborate takeaway: "This study represents a significant step in high-dimensional data inference, improving our understanding of the universe through advanced image reconstruction techniques in astrophysics."

Fine, now please "re-write employing a newspaper-style inverted pyramid structure," we instructed. To which ChatGPT opened with a classic journalistic formulation – "In a groundbreaking astrophysics study ..." – and continued with two long, detailed paragraphs. "Continue," we prompted, and it did, for another long paragraph, so we repeated the command, and it did again.

"Simplify for a Grade 11 audience," we then asked, trying to dumb things down a bit.

"In a recent study," ChatGPT replied, "scientists developed a new way to look at images of distant galaxies that have been warped by the gravity of other galaxies in between. This warping effect is called strong gravitational lensing. The scientists used a special kind of math called score-based modeling to make clearer pictures of these distant galaxies. This is important because it helps us understand things like dark matter and how galaxies form and change. Their method is much better than older ones, giving us clearer and more accurate images. This could lead to new discoveries in space science!"

Maybe a little too simple. How about you "re-write from the beginning, this time for an undergraduate audience," we suggested.

So it did: "In a recent astrophysics study, researchers developed a sophisticated method to analyze images of galaxies that are distorted by the gravitational effect of other galaxies, a phenomenon known as strong gravitational lensing. This distortion can significantly alter our perception of these distant galaxies, making it challenging to study their true properties.

"To address this, the researchers used a technique called score-based modeling, which involves advanced mathematics and stochastic differential equations. This method allows them to create more accurate and clearer images of the distorted galaxies (and) represents a significant leap forward, offering a more reliable way to study and understand the intricate details of distant galaxies and the cosmic forces acting upon them."

Three "Continue ..."s later, we asked ChatGPT to "insert quotes from the lead researchers," but here the software balked. "I couldn't find specific quotes," it replied. "However, if you're interested in exploring the study further or seeking comments from the researchers, I would recommend contacting them directly through their affiliated institutions or looking for interviews and presentations they may have given on this topic." Fair enough.

Then we noticed that nowhere in all the answers ChatGPT had given was our physicist Laurence Perreault-Levasseur or any of her co-authors mentioned, so we asked that it do a re-write incorporating their names and university affiliations, and it did. Finally, after responding to a long series of additional requests – find a better headline sub-head and lead paragraph; use less jargon; attribute some material as direct quotes by the researchers (it refused) – it gave us:

After several more paragraphs of the like, it was time for a re-write by a human being, which is where we came in. Here's the result:

***

Now on to the second study, the one about brain trauma in children.

"Write a news article of 800 words (headline, subhead, paragraph breaks, inverted pyramid style) from the attached study," we asked ChatGPT, providing it with a pdf of Miriam Beauchamp's research.

The response began well enough, if a bit wordily:

And so on. Can you spot the improvements?

We reached out to Perreault-Levasseur, expecting to hear a point of view we sometimes get from researchers faced with "popularized" versions of their work: that the news article would be better if the science hadn't been oversimplified.

Unlike ChatGPT, the astrophysicist's response wasn't instantaneous. "Your emails went to my spam folder," she apologized, adding that she'd been busy travelling for work (a seminar in Princeton, a conference in Waterloo, another in New York).

Yes, she said, she did have some issues with what we sent her.

In general, it lacked a feeling for "the bigger picture ... something we scientists would typically spend most of our time and words explaining when talking to the public."

In particular, she added, "while I think your text is pretty good based on what GPT told you, one thing that's not quite accurate is that we're not really 'de-blurring', we're 'reconstructing' undistorted images of background sources."

There followed a detailed explanation of how that works: she used the analogy that "gravitational lensing is like looking at the flame of a candle through the foot of a wine glass," i.e. there's distortion, and it needs to be "un-done."

Isn't that the same as "de-blurring"? Not quite. It's more a matter of reconstructing what the distorted galaxy looks like, and using artificial intelligence to do so.

In her study, Perreault-Levasseur explained, "we showed how we can use machine learning and deep learning to get a much more complex and realistic representation of galaxies from other images of undistorted galaxies, and then use that to 'fill' the missing information in our reconstructions.

"This allows us not only to do reconstructions that are far better than what existing methods could do, but also, much more importantly, to quantify the uncertainties of our reconstructions ... This is an important step forward in using machine learning to make scientific discoveries in astrophysics."

Then the clincher:

"What's interesting in your using GPT, a deep learning model, to produce a news article is that the whole point of our using deep learning for scientific inference was missed!"

Score one for the human.

After this introduction, the text cycled through another eight long paragraphs, titling each: key findings, parental stress and family burden, in-depth analysis, clinical implications, methodology, limitations and future research, a conclusion, and finally, something short "about the authors."

Take 3: this time with instructions to include details of the study size and authors, ChatGPT provided an even more idiomatic headline and text:

On the fourth and final attempt, we recapitulated our commands thus: Write a news article of 800 words (headline, subhead, para breaks, inverted pyramid style) from the attached study. Plain English, please, avoid jargon and try to use the active voice. Audience are readers with an undergraduate-level education."

After some fine-tuning and revision by us, the final version came out thus:

Not bad, right?

But what did the researchers think? We shared the text with them. Five days later, we had their responses.

"My overall impression?" replied Beauchamp. "This exercise can be useful for quickly understanding the main conclusions of a study, and ChatGPT seems generally good at choosing the right level of language needed for a non-expert audience.

"However, in my opinion there is major confusion between what the authors of the study say (e.g. their interpretation and conclusions in the discussion of the article) and what comes from the existing literature (e.g. what is reported in the introduction)," she continued.

"ChatGPT also takes the liberty of attributing quotes to the authors that aren't real quotes, which seems problematic to me."

(Actually, we had to explain, those quotes weren't ChatGPT's doing, they were ours. Human error? Not quite: they would have been submitted to the researchers for validation before publication, and they would have had the liberty to change them or reject them altogether.)

Beauchamp summed up with some pros and cons:

Pros:

• The general conclusion of the study is fair and formulated in a simple way.
• The methodological details (number of participants, study groups/population) are reported and in sufficient detail for the study to be understood. Note, however, that this information was included only after a prompt to that effect had been added.
• I found the "follow up questions" interesting. When asked to give examples from the questionnaire used in our study, ChatGPT explains that it can't do so because the content of the questionnaire is not included in the study, it is not freely available in the public domain, and it is protected by copyright, which is absolutely true.  I was surprised by this answer, because in other areas, ChatGPT does not state its sources and is not careful about where the information comes from.

Cons:

• For me the biggest problem is that there is a confusion between what is reported in the introduction and what is reported in the discussion. You get the impression that the authors of the article are stating certain things, when actually the conclusions come from studies previously published by other researchers.  ChatGPT does not seem to differentiate between sources of information: what belongs in the existing literature versus what is said by the authors of the article).
• There is no mention of the study limitations, something we are careful to  point out when we speak about our studies in the media. Researchers usually explain the nuances of their work. This is not addressed at all. 
• Although the level of popularization generally seems adequate, ChatGPT does not go any further to explain certain terms, something I would usually do when speaking to a non-expert audience.

Beauchamp's PhD student and the first author of the study, Lara-Kim Huynh, chimed in as well:

• The quotes reported by ChatGPT are problematic, as they're not found in the text of the study. Beauchamp is quoted as saying "Parental differences in rating their child's behavior can be informative," whereas the study clearly attributes that to another author: "De Los Reyes (2013) suggested that parental discrepancies can be informative, rather than be interpreted as measurement error'').
• ChatGPT does a good job of identifying the article's general conclusion. However, the structure of the summary does not respect the structure of the ideas presented in the text. As a result, it's hard to grasp why the authors conducted this study and what the implications are for future research. Instead, ChatGPT seems to report randomly chosen phrases from the text. It rephrases the study's conclusion several times, interspersed with information from the methodology, making it difficult to understand.
• Some sentences are actually incorrect, such as: "Previous research often focused only on mothers' reports and reflected their generally negative perceptions of how their child was doing." ChatGPT seems to have merged these two sentences, to create new information: "Moreover, most studies documenting behavioral changes after pediatric mTBI rely on a sole informant (...)," in the introduction, and "Together, the findings suggest that higher ratings of behavioral problems by mothers compared to fathers on the CBCL are not due to a generally more negative perception of their child's functioning (...)", in the discussion.
• The title "Moms and Dads differ on how a brain injury changes how their child behaves" is very poorly formulated and does not reflect the content of the article.

***

Could ChatGPT – and we who command it – do better next time? It certainly could, our coach, Yann Fortier, explained after taking a close look at how our "conversations" with the software went.

"I really like your sequence of requests, starting at a macro level and moving on down to the micro level," he responded.

But then he had a couple of big, potentially game-changing suggestions:

"Ask it what it thinks would be the best approach for generating a 750-word article, specifying the context, etc. to make its job easier. Your method of repeatedly asking it to "continue" is certainly effective, however, I see an optimization risk.

"In my opinion, ChatGPT doesn't know in advance that it has to 'continue'. So it probably tends to adopt a 'will to conclude', and to do so at the end of each of its replies rather than at the very end of the text.

"That's why, as an example, I prefer an approach like:

"Summarize content X into 10 main sections, numbered and listed from 1 to 10. Step 1 should be the introduction, step 10 the conclusion. Each section should flow logically and continuously."

"Once the results are displayed, ask it to write a 75-word text for step 1. Then, "Continue by writing a 75-word text for step 2." When you get that, say "Same for step 3." And on through steps 4, 5 and 6.

"It's like you're giving it a gradual mini-apprenticeship. And I see two advantages of doing things this way: you'll reach your goal of 750 words in a more direct way, instead of beating around the bush, and ChatGPT will start 'understanding' each step and thereby follow a more logical narrative arc, and will avoid looking for a 'punchline' to end each of the segments.

"I invite you to take the test to see if the answers can prove more accurate," Fortier concluded.

And so we well might. Like ChatGPT itself, our use of it is just beginning.