Information to Wisdom: Commonsense Knowledge Extraction and Compilation

Tutorial at WSDM'21, March 8, 2021, 9am IST

https://www.wsdm-conference.org/2021/tutorials.php#3

Summary

Commonsense knowledge is a foundational cornerstone of artificial intelligence applications. Whereas information extraction and knowledge base construction for instance-oriented assertions, such as Brad Pitt's birth date, or Angelina Jolie's movie awards, has received much attention, commonsense knowledge on general concepts (politicians, bicycles, printers) and activities (eating pizza, fixing printers) has only been tackled recently. In this tutorial we present state-of-the-art methodologies towards the compilation and consolidation of such commonsense knowledge (CSK). We cover text-extraction-based, multi-modal and Transformer-based techniques, with special focus on the issues of web search and ranking, as of relevance to the WSDM community.

Outline

Presenters

• Simon Razniewski is a senior researcher at the Max Planck Institute for Informatics in Saarbrücken, Germany, where he heads the Knowledge Base Construction and Quality research area. He was previously assistant professor at the Free University of Bozen Bolzano (2014-2017), and spent research visits at the University of Queensland, AT&T Labs-Research and the University of California, San Diego. His research includes commonsense knowledge extraction, and he has relevant didactical experience from a lecture on information extraction, and seminars on commonsense knowledge.
• Niket Tandon is a senior research scientist at Allen Institute for Artificial Intelligence, Seattle, WA. His research interests are in commonsense knowledge acquisition and injecting commonsense into neural models for text understanding applications under the Aristo system which aced science exams and had wide press coverage. He completed his PhD from the Max Planck Institute for Informatics in Germany in 2016, that resulted in the largest automatically extracted commonsense knowledge base called WebChild, which has since been used in NLP applications and in Visual Question Answering. He founded an online project mentorship organization, PQRS Research, where he has supervised several undergraduate and graduate theses. He has taught in summer schools on relevant topics and has given a tutorial at CIKM 2017 on ``Commonsense for Machine Intelligence''.
• Aparna Varde is an associate professor of computer science at Montclair State University. Her research spans commonsense knowledge, smart cities, text mining and robotics. She gave tutorials at CIKM 2017 (Commonsense for machine intelligence), DASFAA 2015 (Scalable learning technologies for big data mining) and EDBT 2010 (The hidden web, XML and semantic web). Her tutorials have 2000+ views on SlideShare. Aparna teaches courses such as AI, data mining, machine learning, HCI, database systems and advanced databases. Her tutorial at DASFAA 2009 in the web knowledge discovery area was part of a summer school at QUT, Australia. She was the founder and chair of PIKM (PhD workshop in CIKM) and co-chaired it 5 times. She gave a talk in the Bloomberg Distinguished Speaker Series on commonsense knowledge in the smart cities paradigm. She has 3 best paper awards at IEEE conferences, around 100 publications, and grants from NSF, PSEG etc. She has been a visiting researcher at the Max Planck Institute for Informatics in Saarbrücken, Germany. For details, please see (www.montclair.edu/~vardea).

References

[1] Jo Best. IBM Watson: The inside story of how the Jeopardy-winning supercomputer was born, and what it wants to do next. TechRepublic, 2013.
[2] Sumithra Bhakthavatsalam, Chloe Anastasiades, and Peter Clark. Genericskb: A knowledge base of generic statements. arXiv preprint arXiv:2005.00660, 2020.
[3] Sumithra Bhakthavatsalam, Kyle Richardson, Niket Tandon, and Peter Clark. Do dogs have whiskers? a new knowledge base of haspart relations. arXiv preprint arXiv:2006.07510, 2020.
[4] Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. Comet: Commonsense transformers for automatic knowledge graph construction. arXiv preprint arXiv:1906.05317, 2019.
[5] Yohan Chalier, Simon Razniewski, and Gerhard Weikum. Joint reasoning for multi-faceted commonsense knowledge. AKBC, 2020.
[6] Min Chen, Stefanos Nikolaidis, Harold Soh, David Hsu, and Siddhartha S. Srinivasa. Planning with trust for human-robot collaboration. In ACM/IEEE International Conference on Human-Robot Interaction HRI, pages 307–315, 2018.
[7] Xinlei Chen, Abhinav Shrivastava, and A. Gupta. Neil: Extracting visual knowledge from web data. 2013 IEEE International Conference on Computer Vision, pages 1409–1416, 2013.
[8] Sreyasi Nag Chowdhury, Niket Tandon, Hakan Ferhatosmanoglu, and Gerhard Weikum. VISIR: visual and semantic image label refinement. In ACM WSDM Conference, pages 117–125. ACM, 2018.
[9] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try
arc, the AI2 reasoning challenge. CoRR, abs/1803.05457, 2018.
[10] C. J. Conti, Aparna S. Varde, and Weitian Wang. Task quality optimization in collaborative robotics. In IEEE Big Data, 2020.
[11] Christopher J. Conti, Aparna S. Varde, and Weitian Wang. Robot action planning by commonsense knowledge in human-robot collaborative tasks. In IEEE
International IoT, Electronics and Mechatronics Conference, 2020.
[12] Ernest Davis. Representations of commonsense knowledge. Morgan Kaufmann, 1990.
[13] Ernest Davis. Logical formalizations of commonsense reasoning: A survey. Journal of Artificial Intelligence Research (JAIR), 59:651–723, 2017.
[14] Ernest Davis and Gary Marcus. Commonsense reasoning and commonsense knowledge in artificial intelligence. Communications of the ACM, 2015.
[15] Gerard de Melo, Niket Tandon, and Aparna S. Varde. Commonsense for machine intelligence: Text to knowledge and knowledge to text. In ACM CIKM, 2017.
[16] Kiana Ehsani, Hessam Bagherinezhad, Joseph Redmon, R. Mottaghi, and Ali Farhadi. Who let the dogs out? modeling dog behavior from visual data. 2018
IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4051–4060, 2018.
[17] Yanai Elazar, Abhijit Mahabal, Deepak Ramachandran, Tania Bedrax-Weiss, and Dan Roth. How large are lions? inducing distributions over quantitative attributes.
ACL, 2019.
[18] J. Feldman, Joe Davison, and Alexander M. Rush. Commonsense knowledge mining from pretrained models. EMNLP, abs/1909.00505, 2019.
[19] M. Forbes, Ari Holtzman, and Yejin Choi. Do neural language representations learn physical commonsense? CogSci, abs/1908.02899, 2019.
[20] Anurag Garg, Niket Tandon, and Aparna S. Varde. I am guessing you can’t recognize this: Generating adversarial images for object detection using spatial commonsense (student abstract). In AAAI, pages 13789–13790, 2020.
[21] Shalini Ghosh, Giedrius Burachas, Arijit Ray, and Avi Ziskind. Generating natural language explanations for visual question answering using scene graphs and
visual attention. arXiv preprint arXiv:1902.05715, 2019.
[22] Jonathan Gordon and Benjamin Van Durme. Reporting bias and knowledge acquisition. In AKBC, 2013.
[23] Filip Ilievski, Pedro Szekely, Jingwei Cheng, Fu Zhang, and Ehsan Qasemi. Consolidating commonsense knowledge. arXiv preprint arXiv:2006.06114, 2020.
[24] Filip Ilievski, Pedro Szekely, and Daniel Schwabe. Commonsense knowledge in wikidata. Wikidata workshop at ISWC, 2020.
[25] Maged N. Kamel-Boulos and Estella M. Geraghty. Geographical tracking and mapping of coronavirus disease covid-19/severe acute respiratory syndrome coronavirus 2 (sars-cov-2) epidemic and associated events around the world: how 21st century gis technologies are supporting the global fight against outbreaks and epidemics. International Journal of Health Geographics, 19(8), March 2020.
[26] Divyadharshini Karthikeyan, Aparna S. Varde, and Weitian Wang. Transfer learning for decision support in covid-19 detection from a few images in big data. In IEEE Big Data, 2020.
[27] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123(1):32–73, 2017.
[28] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. CoRR, abs/1908.03557, 2019.
[29] Zhenhui Li, Huaxiu Yao, and Fenglong Ma. Learning with small data. In WSDM, 2020.
[30] Hongyu Lin, Le Sun, and Xianpei Han. Reasoning with heterogeneous knowledge for commonsense machine comprehension. In EMNLP, 2017.
[31] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In European Conference on Computer Vision ECCV, pages 740–755. Springer LNCS, 2014.
[32] MD Mark Mosley. 16 common-sense tips and facts for dealing with covid-19. Emergency Medicine News, 42(4A), April 2020.
[33] Aditya Mogadala, Marimuthu Kalimuthu, and Dietrich Klakow. Trends in integration of vision and language research: A survey of tasks, datasets, and methods. CoRR, abs/1907.09358, 2019.
[34] Tuan-Phong Nguyen, Simon Razniewski, and Gerhard Weikum. Advanced semantics for commonsense knowledge extraction. WWW, 2021.
[35] Janna Omeliyanenko, Albin Zehe, Lena Hettinger, and Andreas Hotho. Lm4kg: Improving common sense knowledge graphs with language models. In ISWC, 2020.
[36] Tulin Ozturk, Muhammed Talo, Eylul Azra Yildirim, Ulas Baran Baloglu, Özal Yildirim, and U. Rajendra Acharya. Automated detection of COVID-19 cases using deep neural networks with x-ray images. Computers in Biology and Medicine, 121:103792, 2020.
[37] Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. Language models as knowledge bases? ACL, 2020.
[38] Julien Romero and Simon Razniewski. Inside Quasimodo: Exploring construction and usage of commonsense knowledge. In CIKM, 2020.
[39] Julien Romero, Simon Razniewski, Koninika Pal, Jeff Z. Pan, Archit Sakhadeo, and Gerhard Weikum. Commonsense properties from query logs and question answering
forums. In CIKM, 2019.
[40] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In AAAI Conference on Artificial Intelligence, pages 8732–8740. AAAI Press, 2020.
[41] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, ACL Volume 1: Long Papers, pages 2556–2565, 2018.
[42] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, R. Mottaghi, Luke Zettlemoyer, and D. Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10737–10746, 2020.
[43] Vered Shwartz, Peter West, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Unsupervised commonsense question answering with self-talk. EMNLP, abs/2004.05483, 2020.
[44] Jakob Suchan and Mehul Bhatt. Commonsense scene semantics for cognitive robotics: Towards grounding embodied visuo-locomotive interactions. In IEEE
Intl. Conf. on Computer Vision (ICCV) workshops, pages 742–750, 2017.
[45] Alon Talmor, Yanai Elazar, Y. Goldberg, and Jonathan Berant. olmpics-on what language model pre-training captures. Transactions of the Association for Computational Linguistics, 8:743–758, 2019.
[46] Niket Tandon, Bhavana Dalvi, Joel Grus, Wen tau Yih, Antoine Bosselut, and Peter Clark. Reasoning about actions and state changes by injecting commonsense
knowledge. ArXiv, abs/1808.10012, 2018.
[47] Niket Tandon, Gerard De Melo, Fabian Suchanek, and Gerhard Weikum. Webchild: Harvesting and organizing commonsense knowledge from the web. In WSDM, 2014.
[48] Niket Tandon, Aparna S. Varde, and Gerard de Melo. Commonsense knowledge in machine intelligence. ACM SIGMOD Record, 46(4):49–52, 2017.
[49] Kkaus-Dieter Thoben, Stefan Wiesner, and Thorsten Wuest. Industrie 4.0 and smart manufacturing - a review of research issues and application examples. Intl. J. of Automation Tech, 11(1):1—-12, 2017.
[50] Cadie Thompson. New details about the fatal tesla autopilot crash reveal the driver’s last minutes. Business Insider, 2017.
[51] Jiangnan Xia, Chen Wu, and Ming Yan. Incorporating relation knowledge into commonsense reading comprehension with multi-task learning. CIKM, 2019.
[52] F. F. Xu, B. Y. Lin, and K. Q. Zhu. Automatic extraction of commonsense located-near knowledge. In ACL conference, pages 96–101, 2018.