An idea alone is like a single node in a network – it may hold potential, but without connection and communication, it remains isolated and unable to realize its full power.

Assessment methods, such as essays, projects, and presentations, are crucial in evaluating students’ mastery of critical thinking, problem-solving, communication skills, and content knowledge. These methods are integral to academic programming and preparing students for success in their future pursuits. This blog post explores the impact of AI models, particularly ChatGPT, on academic programming and assessment methods used to measure academic programming success.

While AI models offer benefits, such as improving students’ communication skills and productivity, there are concerns about potential misuse as a plagiarism mechanism. This post discusses the advantages and drawbacks of AI models and their potential impact on the assessment methods of essays, projects, and presentations. These methods are critical tools for educators to measure students’ mastery of content knowledge, critical thinking, problem-solving, and communication skills. 

AI models, including ChatGPT, may pose potential threats to the integrity of existing measurements, and it is crucial to address these concerns. At the same time, these models may unlock opportunities for students not as skilled in communication. Improving student communication may unlock opportunities for an idea alone is like a single node in a network – it may hold potential. Still, without connection and communication, it remains isolated and unable to realize its full power.

AI Detection Scores Maybe Expanding the Definition of Plagiarism

Plagiarism involves using another person’s ideas or work without proper credit or attribution. It can take various forms, including copying and pasting text, paraphrasing without citing the source, and presenting someone else’s ideas or arguments as one’s own.

With the advent of AI language models such as ChatGPT, the definition of plagiarism has expanded to include using such models to generate essays or written assignments without fully understanding the content or ideas presented. This raises concerns about academic integrity and the value of educational programming, as students may not develop the critical thinking and problem-solving skills essential for success in their future endeavors.

These assessment methods are quality control measures used by academic institutions to identify students who require further assistance and ensure that they meet their academic program’s learning objectives. Failure to meet program learning objectives can have profound implications, such as doctors without sufficient knowledge of anatomy or plumbers who do not understand fractions.

Ironically, search engines like Google also face similar quality assurance concerns, as AI-generated content may lack original thought and creativity and even contain false information.

Do AI Detectors Really Detect Plagiarism?

This new concern over AI-generated content has rapidly formed a new industry dedicated to AI detection. Companies such as Originality.ai, Content at Scale, Copyleak, Writer, and others have formed, touting the ability to detect AI content by providing AI scores.

These programs aim to ensure the originality and creativity of written works by identifying and addressing plagiarism. If these tools are successful, educators can regain control over the assessment process. Students may also be deterred from using AI-generated content to replace genuine effort and work due to fear of being detected and facing the consequences.

While tools like AI-powered plagiarism detectors can effectively detect similarities in words, they may not always discern a lack of original thought or creativity.

Is calling AI generation plagiarism going too far?

It is valid to consider AI-generated content as a form of plagiarism, but it is essential to understand what these AI Detectors are measuring. The nuanced definition of plagiarism may include repeating memorized content without proper citation, which can result from the educational process. This creates a challenge in determining a student’s genuine work, as their education shapes their ideas and thoughts.  As a result, finding a balance between promoting originality and identifying plagiarism is crucial to ensure a fair and effective education system for all students.

Do these new AI detectors really measure original thought?

AI detectors use various methods to score AI-generated content, such as comparing it to known datasets, identifying language usage and patterns, and analyzing text structure and style. Machine learning algorithms also determine if an AI system or a human-generated text. These methods allow educators to identify if AI-generated content has been used and to what extent, ensuring the integrity of academic work.

Before AI-detectors, plagiarism detectors used a variety of techniques, including:

• Manual comparison: Manually comparing a suspect document with known sources to identify similarities in phrasing or ideas.
• Search engine queries: Searching for suspect phrases or passages on search engines such as Google to identify potential sources.
• Electronic comparison: Using software to compare a suspect document with a known source to identify similarities in wording or structure.
• Word frequency analysis: Analyzing the frequency of specific words or phrases in a document to identify potential sources.
• Fingerprinting: Creating a unique “fingerprint” of a document based on its content and comparing it against other documents to identify potential matches.

Albeit not perfect, these techniques had specificity and seemed to have a reasonably strong correlation to using someone else’s thoughts.

AI detectors are still evolving, but these are some of the dominant techniques they are using for AI-generated scoring:

• Perplexity: This measures how well a language model can predict the next word in a sequence of words. The lower the perplexity score, the more accurate the language model is considered to be.
• Burstiness: This refers to the level of coherence and consistency in the writing style of a piece of content. An AI detection program may flag unusually coherent or inconsistent content with the author’s previous work.
• N-gram analysis: This technique involves analyzing sequences of N words in a piece of content to identify patterns that may indicate plagiarism.
• Synonym detection: AI detection programs may compare the vocabulary used in a piece of content to known sources to detect instances of synonym substitution.
• Source code analysis: Some AI detection programs may examine the underlying code of an AI-generated text to detect patterns that suggest plagiarism.
• Machine learning: Some AI detection programs use machine learning algorithms to identify patterns in text and assign a score based on the likelihood of plagiarism.

Integrating traditional plagiarism detectors with newer AI detectors could give educators more confidence in accurately identifying plagiarism. However, relying solely on AI-detector scores to accuse a student of plagiarism is not dependable. The detectors lack independent validation, and the weighting of different scoring techniques remains uncertain. Despite this, AI-detectors hold potential for the future, such as improving the detection of paraphrasing, an area where traditional detectors are not as proficient.

For example, paraphrasing involves rephrasing someone else’s ideas or work using one’s own words, while plagiarism involves using someone else’s work or ideas without proper credit. N-gram analysis offers potential in detecting paraphrasing by breaking down the text into sequences of n words and examining the frequency and distribution of these n-grams within the text.

My Experiment with AI-Detectors Uncovers Limitations

In this section, I will discuss my experiments with AI detectors. While these tests are not scientific or statistically significant, they aim to stimulate discussion on the topic and encourage continuous improvement efforts in developing a quality assurance tool for academia. As my science and statistics professors would say, all significant experiments begin with small pre-experimental tests.

Consistent Writing Technique is Not an Indicator of Content Originality

I compared the AI detector scores of my original book content and my editor’s changes during the manuscript review process. Surprisingly, the editor’s content had a lower AI score than my original material. I explored this using the AI detector techniques of perplexity and burstiness. 

Perplexity measures a language model’s ability to predict the next word in a text. With their expertise and training, professional writers may have a lower perplexity score, leading to more accurate predictions by the language model. On the other hand, novice writers like me may have a more unpredictable writing style, resulting in a higher perplexity score. 

Burstiness, another measure used in AI plagiarism detectors, refers to the coherence and consistency of a piece of content’s writing style. A professional writer is likelier to have a lower burstiness score due to their consistent and coherent writing style. In contrast, novice writers may have a higher burstiness score due to their inconsistent and varied writing style.

Using AI to Improve Original Content Can Increase AI Detector Scores

I conducted an experiment where I tested a random chapter from my published textbook on different detectors, including a traditional plagiarism detector, an AI detector, and ChatGPT. I am only discussing my preferred AI detector, Originality.ai, for simplicity.

• My first step was to apply a traditional plagiarism detector to the chapter, and it successfully detected my book as a reference.
• Next, I used an AI detector that showed less than 2% of AI content generation, implying that the chapter was primarily based on my original ideas.
• After that, I ran the same chapter through ChatGPT and Jasper.ai, and their outputs were placed in the AI detector. The AI-generated content scores were 93% and 97%, respectively, indicating that only 3% to 7% of my content was considered original. This highlights the potential issues with solely relying on AI-powered plagiarism detectors for originality.

Although my experiment’s results were not statistically significant, they raise vital questions about using AI-powered plagiarism detectors in academia. 

In this situation, AI was used to improve the clarity of original content resulting in a high AI content detector score. It did indeed detect the use of AI; however, it is false that this score can be used as a surrogate for originality. 

Finding the Balance Between Originality and Effective Communication 

Overcoming the Fear of Writing: How AI Tools Can Boost Confidence and Productivity

Various artists, including Johnny Cash’s song, Hurt, Amanda Gorman’s inauguration poem “The Hill We Climb,” and most recently Ren’s song “Hi Ren.”, can take thoughts and emotions and communicate them powerfully through their art. This talent is found in the realm of the arts and among journalists and writers.

My wife is a fantastic writer, and I am in awe of her ability to communicate even the most complex ideas. In fact, I dedicated my first book to her, stating that the book would be unreadable without her talent. However, not everyone possesses the same level of communication skills, which can create fear and anxiety around writing.

We All Do Not Possess the Communication Skills of the Writer, the Journalist, or the Artist

As much as we admire the creativity and mastery of professional writers, journalists, and artists, not everyone possesses their level of communication skills. Even with years of practice, some individuals may still struggle to articulate their thoughts and emotions coherently and effectively.

Writing proficiency is more than a mere skill – it is the key to unlocking one’s potential. Without the ability to communicate ideas effectively, fear and anxiety can take hold, limiting opportunities and isolating even the most brilliant ideas from the rest of the world.

Fortunately, AI-powered writing tools, such as ChatGPT and Jasper, can help bridge this gap by providing a writing partner and offering real-time suggestions for improvement. These tools can help individuals improve their writing skills and gain confidence in expressing themselves effectively.

Moreover, AI writing tools can offer a level playing field for individuals who may not have the same opportunities for education or training in communication skills. For instance, STEM students may excel in technical and analytical skills but struggle with writing, while non-native English speakers may face additional language barriers. AI writing tools can help level the playing field and allow individuals from diverse backgrounds to communicate their ideas with clarity and impact.

Conclusion: More Work Needs to be Done    

The emergence of AI models has brought significant changes to academic programming and assessment methods, particularly in the realm of plagiarism detection. While AI-powered tools can offer valuable support in academic writing, there is a need to balance the promotion of originality and effective communication skills with preserving academic integrity.

As educators and learners, we must continually reflect on the role of assessment methods and AI-powered tools in achieving academic goals. Critical questions regarding the effectiveness and reliability of AI-powered writing tools and plagiarism detectors must be asked. The nuances of plagiarism and originality in the AI era must be considered. We must also recognize the limitations of AI models and the importance of continued improvements and validation to ensure accuracy and effectiveness.

We call on educators and academic institutions to explore the potential of AI-powered tools while maintaining a critical approach and developing ethical guidelines to ensure academic integrity and promote originality and effective communication skills. Students should also focus on developing their critical thinking, problem-solving, writing, and communication skills to succeed in their future endeavors and contribute to a more equitable and innovative society.