On March 11, the ChatGPT and Large Model Symposium sponsored by CAAI was successfully held in Beijing. The symposium was organized by CAAI Special Committee on Natural Language Understanding, Data Grand, and ZhenFund, and supported by the Cloud Computing and Big Data Research Institute of China Academy of Information and Communication Technology (CAICT). Centering on the development and application of ChatGPT and large language models (LLM), many AI experts and scholars gathered together and discussed the cutting-edge technology and the future of AI industry, presenting a wonderful feast of idea exchange. The symposium was broadcast live on five platforms, attracting more than 100,000 viewers offline and online.

In his welcome address, Ma Shaoping, vice chief supervisor of CAAI and professor at Tsinghua University, expressed CAAI’s hopes to conduct in-depth cooperation with AI experts and industrial partners, keep pace with the times, and jointly promote the sustainable development of China’s LLM industry through continuous innovation and investment.

He Baohong, director of the Cloud Computing and Big Data Research Institute of CAICT addressed at the symposium. He said that ChatGPT has brought a new wave of development of AI technology and application. Although there is still a long way to go before reaching Artificial General Intelligence (AGI), we should realize that we are on the right path to use AI technology for analysis, understanding, and generation. This is both an opportunity and a challenge for Chinese and global AI enterprises. All parties should work jointly to accelerate the development of AI industry.

During the presentation session, many AI experts shared their views on the academic development, application, and prospect of ChatGPT and LLM, including Qiu Xipeng, professor at Fudan University School of Computer Science and head of the MOSS System, Zhang Jiajun, research fellow at Chinese Academy of Sciences Institute of Automation, Wan Xiaojun, secretary-general of CCF Special Committee on Natural Language Processing and professor at Peking University, Chen Yunwen, chairman and CEO of Data Grand and doctor of Computer Science at Fudan University, and Zhang Junlin, director of New Technology R&D at Sina and director of CIPS.

According to Professor Qiu Xipeng, in the next few years, ChatGPT will not simply take the form of Chat. The LLM behind it may be integrated into our lives in broader forms. In his presentation, Professor Qiu introduced three key technologies of ChatGPT: situational learning, chain of thought, and command learning. Although ChatGPT still has its limitations, such as the current form being a language model, not controllable, and not connected to the real world, it gives a clear direction to the research development of AGI, and it is very important to keep the model credible, helpful, harmless, and honest in the process.

Zhang Jiajun shared his thoughts on ChatGPT by offering a brief review of technology, and then talked about prompt learning, command learning, and related explorations.

According to Dr. Zhang, there are two main directions for large models, one is “pre-training plus fine-tuning”, which is to fine-tune the large model for downstream tasks, and then a large model for downstream tasks is generated, and the other is “pre-training plus prompt learning”, where pre-training is followed by prompt learning to motivate the large model to complete a specific task. It has been shown that learning prompts is very effective for improving model performance. Therefore, how to learn or find prompt is crucial.

Professor Wan Xiaojun discussed several issues of natural language generation evaluation. He believes that evaluation is a beacon to the development of technology as well as a goal for model optimization. Currently, the human evaluation and automatic evaluation of NLG are still facing problems in terms of fairness, repeatability, and low cost. Using ChatGPT to do automatic evaluation of NLG can yield much better results than do the previous models,  which brings hope for the effect of automatic evaluation.

Dr. Chen Yunwen shared Data Grand’s exploration of LLM in vertical domains, including the exploration of parameters and dataset sizes, adaptative pre-training in vertical domains, the exploration of fine-tuning, prompt engineering and optimization, model training acceleration, and the enhancement of model functions in vertical domains, etc. He believes that it is of great significance for both the commercialization and research of LLM to deepen the application of LLM in vertical domains and integrate LLM into the actual business of enterprises. The model developed by Data Grand for vertical domains is named “Cao Zhi”, which is based on the literary quotation of Cao Zhi completing a poem in seven steps. It is hoped that it will be used specifically and vertically as a domestic GPT model to empower various industries.

Zhang Junlin addressed some doubts about the emergent abilities of LLM. A large complex model is composed of many small-scale models. When the number of small-scale models reaches a certain amount, an unpredictable phenomenon might appear in larger models. This is called “emergent phenomena”. According to Zhang, based on current findings, the emergent abilities of LLM do exist, which gives us positive expectations for the future development of LLM. It also indicates that if we keep expanding the range of LLM, the effect of many tasks could be suddenly improved.

In the roundtable dialogue, Ma Shaoping, vice chief supervisor of CAAI and professor at Tsinghua University, Zhou Ming, founder and CEO of Langboat and vice chairman of CCF,  Zong Chengqing, research fellow at Chinese Academy of Sciences Institute of Automation, Dai Yusen, managing partner of ZhenFund, Yang Hao, Huawei AI scientist and doctor of Beijing University of Posts and Telecommunications, and Cao Feng, deputy director of AI Department of the Cloud Computing and Big Data Research Institute of CAICT, exchanged views on the present and future development of domestic “ChatGPT” and large model research.

During the discussion, experts reached a consensus on the development of domestic “ChatGPT” and large models – the gap between Chinese and oversea companies is much smaller in the field of NLP than that in other fields. There is no need to deify ChatGPT. The gap we are facing now is not uncrossable. We need to give domestic models some time to catch up with and outperform ChatGPT. In terms of technology implementation, experts at the symposium hold the view that ChatGPT has accelerated the upstream and downstream development of NLP and the development of chips. To some extent, large models will probably become next-generation infrastructure, and China needs to have its own foundation model system to ensure security, concurrence, and stability. It is hoped that investors, researchers, and AI industry should stay calm, keep focused, and make solid achievements.