This project will investigate the accuracy and biases of ChatGPT in identifying climate change hazards, comparing its responses to IPCC reports. Using a standard prompt for 191 countries, it will analyse disparities using statistical methods and cross-references with World Bank data, enhancing research skills in AI evaluation in environmental science.
Department: Geography & Environmental Science
Supervised by: Hong Yang
This project aims to analyse the accuracy and potential biases of generative AI tools, specifically ChatGPT, in the context of climate change-related hazards. Focusing on enhancing climate literacy, it seeks to compare ChatGPT's responses on climate vulnerabilities with authoritative sources like the International Panel on Climate Change (IPCC) reports. The research will adopt an exploratory approach, formulating a standard prompt template to inquire about each of the 191 IPCC member countries' susceptibility to climate hazards, such as, "What climate change-related hazards is [Country] most vulnerable to?" ChatGPT's responses will be systematically collated and analysed. This will involve cross-referencing the data with the World Bank's country-level income and regional information to identify any disparities in responses based on these parameters. Statistical tools like T-tests and ANOVA will be employed for this purpose. Further, the study will utilise the Index for Risk Management (INFORM) Global Risk Index from the IPCC's AR6 for a detailed comparison of ChatGPT's answers against established data on major climate hazards like floods, droughts, and cyclones. A confusion matrix will be created to visualise discrepancies, with a Chi-squared test of independence applied for statistical validation. This project stands out as it not only allows the student to develop research skills in data analysis and interpretation but also offers exposure to various stages of research, from data collection to statistical testing. The clear structure of tasks, from developing prompts to conducting statistical analyses, ensures a comprehensive learning experience in evaluating AI tools in environmental science.
In this project, the student will engage in a series of structured tasks, each designed to enhance their research skills and understanding of the research process: 1) Developing and applying a set of standardised ChatGPT prompts to assess the climate change vulnerability of 191 countries. This task will hone the student’s ability to frame research questions effectively. 2) Analysing ChatGPT-generated responses using a World Bank dataset for country-level income and regional information. The student will conduct T-tests and ANOVA analyses to explore variations in responses related to income levels and regions, developing their skills in statistical analysis and data interpretation. 3) Utilising the INFORM Global Risk Index from the latest IPCC Assessment Report (AR6) to compare against ChatGPT's responses on key climate hazards such as floods, droughts, and cyclones. This will involve critical evaluation and comparison skills. 4) Creating a confusion matrix to compare IPCC data with AI responses and conducting a Chi-squared test for statistical validation. This task will deepen the student’s understanding of data validation techniques. 5) Compiling a comprehensive project report, which will enable the student to develop their writing and reporting skills. Additionally, there will be opportunities to contribute to an academic publication, enhancing their experience in scholarly communication. The project also offers the chance for the student to present findings at conferences and to participate in the project’s communication and media planning, providing a well-rounded exposure to the diverse aspects of academic research.
Essential Skills and Knowledge: 1) Basic familiarity with generative AI tools like ChatGPT and an understanding of IPCC reports, to effectively engage with the project’s subject matter. 2) Organisational skills with an ability to prioritise tasks to ensure timely and specification-compliant delivery, essential for managing the various stages of the research process. 3) A proactive approach to learning and adopting new research techniques, crucial for the exploratory nature of this project. 4) Competence in reviewing and synthesising literature, aiding in the establishment of a robust theoretical framework. 5) Proficiency in IT, particularly Microsoft Office, for efficient data management and presentation. 6) Solid quantitative skills, with programming experience in R, to carry out statistical analyses. 7) Strong oral and written communication skills for effective dissemination of research findings, both within the academic community and to wider audiences. Desirable Skills: 1) Prior experience in planning and executing small-scale research projects, which will be beneficial for independently managing various components of the project. 2) Basic project management skills, aiding in the structured and efficient execution of the research.
Through this placement, the student will develop a diverse set of skills, applicable in both research-oriented and broader professional contexts: 1) Critical Analysis of Scientific Literature: The student will enhance their ability to critically read and analyse scientific literature, particularly relating to climate change and AI, fostering a deeper understanding of the research area. 2) Advanced Communication Skills: They will refine their communication skills, learning to write and revise ChatGPT prompts to effectively meet research objectives, an essential skill for precise information elicitation. 3) Data Comprehension: The placement will enhance the student’s capability to comprehend complex datasets from authoritative sources like the IPCC and World Bank, a key skill in data-driven research fields. 4) Statistical Analysis Proficiency: The student will gain hands-on experience in statistical analysis, including conducting T-tests, ANOVA, and Chi-squared tests, using R or similar software. This will develop their ability to interpret data sets and draw meaningful conclusions. 5) Data Presentation and Interpretation: The student will learn to interpret and present data in a coherent and academically rigorous manner, a crucial skill for any research endeavour. Additionally, this project offers the potential for the student to contribute to a publication or poster presentation, providing a platform to showcase their research findings and enhancing their experience in academic dissemination.
The placement will be based in the School of Archaeology, Geography and Environmental Science.
Ideally, the placement would be on a full-time basis, amounting to approximately 35 hours per week. However, the project offers flexibility in terms of working hours to accommodate the student's schedule and commitments.
Monday 15 July 2024 - Friday 13 September 2024
The deadline to apply for this project is 5pm on Friday 5th April 2024. To make an application, please go to the following link and complete the application form: https://forms.office.com/e/pMgea0dAHv. To find this project in the application form, please filter ‘school of project applying to’ and select School of Archaeology, Geography & Environmental Science