Article type
Abstract
"Background: The adoption of generative AI, like ChatGPT, in government services is emerging as a solution to improve efficiency and accessibility in the public sector. These technologies promise to offset operational costs and extend service availability beyond traditional office hours. However, the practical effectiveness of AI in meeting specific citizen needs, such as handling local service inquiries, remains under-explored.
Objectives: The study evaluates the effectiveness and limitations of generative AI based chatbot in a municipal setting. To narrow down the scope, we focused on the questions related to daycare application and admission.
Methods: A prototype chatbot was developed, utilizing public documents from the city government's website to ensure accurate responses and avoid “hallucination”. Evaluation was twofold: subjective assessment by municipal office visitors (N=25) and an objective evaluation using a set of pre-collected questions and answers from municipal employees.
Results: Initially, only a quarter of participants were familiar with AI chatbots and had modest expectations. Some had difficulty navigating the chatbot's interface, especially when formulating open-ended questions to elicit desired responses. Despite these initial hurdles, more than half of the users ultimately found the chatbot useful. However, the objective evaluation revealed that the chatbot's ability to accurately answer questions was limited, primarily due to the limited amount of information in the documents provided for response generation.
Conclusions: Generative AI chatbots hold significant potential to enhance municipal services, yet their effectiveness is currently hindered by a lack of comprehensive, relevant input data. Bridging the gap between the specific needs of citizens and readily available digital information and is critical. Addressing this gap is essential to fully harnessing the power of AI technologies to revolutionize city services.
Relevance and Importance: The study highlights the gap between technological capabilities and the infrastructure needed to realize their full potential. It identifies the strengths and weaknesses of such technologies and the steps needed to address these challenges.
Public Involvement: Volunteers were recruited from the municipal office visitors, and municipal employees were interviewed to gather common questions for the chatbot's accuracy evaluation.
"
Objectives: The study evaluates the effectiveness and limitations of generative AI based chatbot in a municipal setting. To narrow down the scope, we focused on the questions related to daycare application and admission.
Methods: A prototype chatbot was developed, utilizing public documents from the city government's website to ensure accurate responses and avoid “hallucination”. Evaluation was twofold: subjective assessment by municipal office visitors (N=25) and an objective evaluation using a set of pre-collected questions and answers from municipal employees.
Results: Initially, only a quarter of participants were familiar with AI chatbots and had modest expectations. Some had difficulty navigating the chatbot's interface, especially when formulating open-ended questions to elicit desired responses. Despite these initial hurdles, more than half of the users ultimately found the chatbot useful. However, the objective evaluation revealed that the chatbot's ability to accurately answer questions was limited, primarily due to the limited amount of information in the documents provided for response generation.
Conclusions: Generative AI chatbots hold significant potential to enhance municipal services, yet their effectiveness is currently hindered by a lack of comprehensive, relevant input data. Bridging the gap between the specific needs of citizens and readily available digital information and is critical. Addressing this gap is essential to fully harnessing the power of AI technologies to revolutionize city services.
Relevance and Importance: The study highlights the gap between technological capabilities and the infrastructure needed to realize their full potential. It identifies the strengths and weaknesses of such technologies and the steps needed to address these challenges.
Public Involvement: Volunteers were recruited from the municipal office visitors, and municipal employees were interviewed to gather common questions for the chatbot's accuracy evaluation.
"