ChatGPT vs. Gemini: AI Writing Styles

Each AI chatbot possesses its own distinctive writing style, much like humans do. Linguists have found that ChatGPT and Gemini AI write using different idioms.

It might seem that you are conversing with multiple individuals when interacting with ChatGPT, and the chatbot may not always appear to have a consistent personality.

Linguists understand that individuals express themselves differently based on factors such as native language, age, gender, and education, which is referred to as an “idiolect,” similar to a dialect but more specific.

The language produced by ChatGPT can be analyzed to determine if it expresses itself in a single, distinct way. Idiolects are crucial in forensic linguistics, which involves analyzing language in police interviews, attributing authorship, tracing linguistic backgrounds, and detecting plagiarism.

There are concerns about LLMs being used by students to outsource writing assignments; therefore, it was determined whether ChatGPT, Gemini, and Copilot possess idiolects.

To test if a text was generated by an LLM, both the content and language used must be examined. ChatGPT tends to use standard grammar and academic expressions, avoiding slang or colloquialisms. It also overuses sophisticated verbs and adjectives.

Comparing ChatGPT and Gemini

A dataset compiled by computer scientist Muhammad Naveed contains hundreds of short texts on diabetes written by ChatGPT and Gemini. The texts are similar in size and can be used to compare the AI models' abilities to generate informative content on a medical topic. The similarities allow a determination of whether the outputs come from distinct “authors.”

The Delta method, introduced in 2001 by John Burrows, compares the frequencies of commonly used words to attribute authorship. It captures features that vary according to authors’ idiolects and measures the linguistic “distances” between texts. A smaller distance indicates a higher likelihood that the author is the same.

A random sample of ChatGPT-generated texts on diabetes had a distance of 0.92 to the entire ChatGPT diabetes dataset and a distance of 1.49 to the entire Gemini dataset. A random sample of Gemini texts had a distance of 0.84 to Gemini and 1.45 to ChatGPT. The authorship is clear, indicating that the tools have distinct writing styles.

Trigram Analysis

By looking at the diabetes texts and selecting words in groups of three, known as “trigrams,” one can get a sense of someone’s unique way of putting words together. The 20 most frequent trigrams for ChatGPT and Gemini were extracted and compared.

ChatGPT’s trigrams suggest a formal, clinical, and academic idiolect, using phrases such as “individuals with diabetes,” “blood glucose levels,” and “an increased risk.” Gemini’s trigrams are more conversational and explanatory, with phrases such as “the way for,” “high blood sugar,” and “blood sugar control.” The choice of “sugar” instead of “glucose” indicates a preference for simple language.

Gemini uses “blood glucose levels” infrequently, while ChatGPT uses “high blood sugar” less often than Gemini. ChatGPT uses “glucose” more than twice as often as it uses “sugar,” while Gemini does the opposite.

Why LLMs Develop Idiolects

The phenomenon could be associated with the principle of least effort, where models continue to use words or phrases that become part of their linguistic repertoire. It might also be a form of priming, where each model is priming itself with words it uses repeatedly.

Idiolects in LLMs may also reflect emergent abilities, skills that the models were not explicitly trained to perform but demonstrate nonetheless. The fact that LLM-based tools produce different idiolects matters for the debate about AI achieving human-level intelligence.

Knowing that LLMs write in idiolects could help determine if an essay was produced by a model or a particular individual, similar to recognizing a friend’s message by their signature style.