In a paper submitted to the Arxiv* server, researchers examined the disparities in user behavior when utilizing chatbot tools and search engines for tasks related to seeking information. A randomized online experiment was conducted, wherein participants were divided into two groups. One group utilized a tool like ChatGPT, while the other group used a tool like Google Search. The results of this study indicate that the ChatGPT group reliably exhibits reduced task completion time across all tasks, while no statistically significant disparity in overall task performance is observed between the groups.
Study: ChatGPT vs. Google Search: Unveiling User Experience and Search Performance. Image credit: NicoElNino / Shutterstock
*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information.
Background
The increasing popularity of ChatGPT highlights the significant transformative capabilities of generative artificial intelligence across diverse sectors. Recently, in February, Microsoft integrated ChatGPT into its Bing search engine, thereby merging the capabilities of chat and search distinctively.
Bing saw a 15.8% traffic gain after a month of this integration. Google's traffic dropped by 1% during this period. According to Yahoo Finance, every 1% of the search promotion market stake generates an annual revenue of $2 billion. To this end, it is essential to note that there are notable distinctions between conventional search engines and systems like ChatGPT in terms of their methodologies for information retrieval.
Google utilizes keyword search and matching algorithms to provide users with a curated selection of pertinent links. In contrast, ChatGPT utilizes a conversational methodology, allowing users to present inquiries using natural language. Although Google's speed is undeniably impressive, users must manually sift through search results, a process that can be pretty time-consuming.
ChatGPT aims to understand the user's intentions and deliver well-structured responses in coherent and grammatically complete sentences. Although it is more user-friendly and spontaneous, it is essential to acknowledge that ChatGPT does have certain limitations. These include longer response times and the potential for providing inaccurate or deceptive information. This contrasts conventional search engines, which typically offer quicker response times and more reliable outcomes.
Methodology
This study carried out a randomized online experiment to comprehensively compare users' search performance and behaviors when using large language model (LLM)-driven chatbots (ChatGPT) and search engine tools (Google Search). It addressed inquiries about specific distinctions in user behaviors between the utilization of ChatGPT and Google Search for information-related tasks and whether these variations exhibit consistency across various types of search tasks, and user search performance across diverse educational backgrounds.
The researchers proposed a new experimental design where they manipulated the task complexity. In addition, they integrated widely employed metrics to evaluate the effectiveness of search functionality and user satisfaction. These metrics encompass factors such as the duration of search tasks, task completion rates, quality of information, and overall satisfaction.
A between-subjects design was utilized in this online experiment, consisting of the LLM powered chatbot condition and the traditional search engine condition. Each participant was furnished with a pre-registered user account to observe and analyze user search behavior.
The ChatGPT tool maintains a log of every user prompt and its corresponding response generated by the GPT-3.5 model. The Google Search tool monitors and records various user interactions, including submitting search queries, viewing search results pages corresponding to those queries, and selecting specific search result items through clicks. The temporal data of both user and application programming interface (API) actions are also documented.
Major Findings
Based on self-reported findings, the average time required for participants in the ChatGPT group to complete the tasks is 11.35 minutes, whereas participants of the Google Search group take 18.75 minutes (representing a 65.20% increase). The ChatGPT group consistently exhibited a lower time allocation for each task compared to the Google Search group across all tasks. All these observed differences exhibit statistical significance at the 1% level.
In addition, the analysis of the server logs for both search tools objectively calculated the duration of each task. The ChatGPT group measures the search duration by calculating the time interval between the user's initial query and the receipt of the final response from the ChatGPT API. The findings indicate a consistent trend where the duration of engagement with ChatGPT is significantly shorter compared to Google Search.
ChatGPT's search effectiveness was found to be lower than that of Google Search because these platforms are used to obtain information differently. While individuals must independently create search queries and experiment to discover the most appropriate results while using Google Search, ChatGPT simplifies searches by letting users ask questions in natural language.
Moreover, in relation to the length of queries, this research indicates that users of ChatGPT tend to generate notably lengthier queries when engaging in search tasks. This is because ChatGPT is specifically designed to facilitate natural language interactions with users.
Furthermore, there is no significant variation in the ChatGPT group's performance based on the participants' educational backgrounds. In contrast, a positive correlation exists between users' education levels and their performance with Google Search. Individuals who possess higher levels of education exhibit enhanced aptitude in utilizing Google Search to rectify errors in the fact-checking endeavor.
Conclusion
The implementation of ChatGPT has demonstrated significant improvements in work productivity, allowing users to complete tasks more efficiently and potentially enhancing the overall user experience. However, it is crucial to acknowledge that ChatGPT does not surpass conventional search engines in terms of performance.
The results of this study provide additional evidence to substantiate the notion that individuals perceive the information produced by ChatGPT to possess superior quality and greater accessibility compared to Google Search. Moreover, respondents hold a comparable level of trust in the outcomes yielded by both platforms. Although ChatGPT users reported considerably better user experience in terms of enjoyment, usefulness, and satisfaction, ChatGPT may result in overdependency and can generate misinformation and offer inconsistent results. Overall, this study provides valuable insights into search engine management and underlines avenues for integrating chatbots into search engines.
*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as conclusive, guide clinical practice/health-related behavior, or treated as established information.
ChatGPT-4: Human-like Behavior in AI Chatbots