Topics in Neural Information Retrieval

• Type: Advanced lecture
• Lecturer: Dr. Andrew Yates
• Credits: 6 ECTS credits
• Time: Tuesdays (and a few Thursdays), 14-16:00 in Room 029 E1.5
• Mailing list for discussion and announcements
• Please register for the mailing list if you'd like to take the class!

Summary

In this course we will be investigating advanced topics in Information Retrieval, with a focus on neural network methods and how they contrast with prior work. In lecture we'll work to understand important problems in IR and how they're solved by both neural and traditional methods, and how these methods relate to IR theory. In contrast with the common claim that neural models are difficult to make sense of, we'll explore how the design of neural IR architectures often follows directly from theory. Assignments will require gaining an in-depth understanding of several related methods by reading scientific reports and demonstrating this understanding by writing an essay analyzing the methods. While beneficial, no background in Deep Learning will be assumed; we'll start with overviews of both IR and DL before moving on to more advanced topics like state-of-the-art retrieval models for assessing the relevance of a document to a given query, diversifying search results based on their novelty (with respect to each other), training neural models with weak supervision, and applications of neural IR to other tasks.

Objectives

By the end of the course, students will be able to describe and contrast state-of-the-art traditional and neural IR approaches, to examine and critique the assumptions made by these approaches, and to critically read and analyze relevant scientific literature.

Prerequisites

Students should have a basic knowledge of Machine Learning. Prior knowledge of Information Retrieval and Neural Networks will be helpful but is not required.

Each class has several related readings. It is recommended that you read the required readings before lecture, supplementing them with the background readings as needed. It is not necessary to read the optional readings before class, though students will read many of them as part of the class assigments. 

References

[1] Christopher D. Manning, Manning Raghavan, & Manning Schütze. 2008. Introduction to Information Retrieval. [chapter pdfs]

[2] Hui Fan, Tao Tao, & ChengZiang Zhai. 2004. A formal study of information retrieval heuristics. Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval (SIGIR '04). [pdf]

[3] Andriy Burkov. The Hundred-Page Machine Learning Book. 2019. [chapter pdfs]

[4] Jiafeng Guo, Yixing Fan, Qingyao Ai, and W. Bruce Croft. 2016. A Deep Relevance Matching Model for Ad-hoc Retrieval. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management (CIKM '16). [pdf]

[5] Chenyan Xiong, Zhuyun Dai, Jamie Callan, Zhiyuan Liu, and Russell Power. 2017. End-to-End Neural Ad-hoc Ranking with Kernel Pooling. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '17). [pdf]

[6] Donald Metzler and W. Bruce Croft. 2005. A Markov random field model for term dependencies. In Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval (SIGIR '05). [pdf]

[7] Donald Metzler. 2011. A Feature-Centric View of Information Retrieval. [chapter pdfs] (link only works from uni network)

[8] Zhuyun Dai, Chenyan Xiong, Jamie Callan, and Zhiyuan Liu. 2018. Convolutional Neural Networks for Soft-Matching N-Grams in Ad-hoc Search. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining (WSDM '18). [pdf]

[9] Kai Hui, Andrew Yates, Klaus Berberich, Gerard de Melo. 2017. PACRR: A Position-Aware Neural IR Model for Relevance Matching. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP '17). [pdf]

[10] Andrew Yates, Kai Hui. 2017. DE-PACRR: Exploring Layers Inside the PACRR Model. In the SIGIR 2017 Workshop on Neural Information Retrieval (NeuIR '17). [pdf]

[11] Kai Hui, Andrew Yates, Klaus Berberich, Gerard de Melo. 2018. Co-PACRR: A Context-Aware Neural IR Model for Ad-hoc Retrieval. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining (WSDM '18). [pdf]

[12] Samuel Huston and W. Bruce Croft. 2014. A Comparison of Retrieval Models using Term Dependencies. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management (CIKM '14). [pdf]

[13] Tao Tao and ChengXiang Zhai. 2007. An exploration of proximity measures in information retrieval. In Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval (SIGIR '07). [pdf]

[14] Chris Olah. 2015. Understanding LSTM Networks. Blog post. [link]

[15] Yixing Fan, Jiafeng Guo, Yanyan Lan, Jun Xu, Chengxiang Zhai, and Xueqi Cheng. 2018. Modeling Diverse Relevance Patterns in Ad-hoc Retrieval. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR '18). [pdf]

[16] Zhiwen Tang and Grace Hui Yang. 2019. DeepTileBars: Visualizing Term Distribution for Neural Information Retrieval. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence (AAAI-19). [pdf]

[17] Michael Bendersky and Oren Kurland. 2008. Utilizing Passage-Based Language Models for Document Retrieval. In Proceedings of ECIR 2008. [pdf]

[18] Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu, and Xueqi Cheng. 2016. A Study of MatchPyramid Models on Ad-hoc Retrieval. In the SIGIR 2016 Workshop on Neural Information Retrieval (NeuIR '16). [pdf]

[19] Probabilistic Latent Semantic Indexing. Thomas Hofmann. SIGIR '99. [pdf]

[20] Information Retrieval as Statistical Translation. Adam Berger and John Lafferty. SIGIR '99. [pdf]

[21] Semantic Term Matching in Axiomatic Approaches to Information Retrieval. Hui Fang and ChengXiang Zhai. SIGIR '06. [pdf]

[22] Clickthrough-Based Translation Models for Web Search: from Word Models to Phrase Models. Jianfeng Gao, Xiaodong He, and Jian-Yun Nie. CIKM '10. [pdf]

[23] Deep contextualized word representations. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. NAACL '18. [pdf]

[24] Attention Is All You Need. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. NIPS '17. [pdf]

[25] BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. NAACL '19. [pdf]

[26] Learning Deep Structured Semantic Models for Web Search using Clickthrough Data. Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, Larry Heck. CIKM '13. [pdf]

[27] Passage Re-ranking with BERT. Rodrigo Nogueira and Kyunghyun Cho. arXiv 2019. [pdf]

[28] Simple Applications of BERT for Ad Hoc Document Retrieval. Wei Yang, Haotian Zhang, and Jimmy Lin. arXiv 2019. [pdf]

[29] CEDR: Contextualized Embeddings for Document Ranking. Sean MacAvaney, Andrew Yates, Arman Cohan, and Nazli Goharian. SIGIR '19. [pdf]

[30] Relevance-Based Language Models. Victor Lavrenko and W. Bruce Croft. SIGIR '01. [pdf]

[31] Learning to Match Using Local and Distributed Representations of Text for Web Search. Bhaskar Mitra, Fernando Diaz, Nick Craswell. WWW '17. [pdf]

[32] Simple Attention-Based Representation Learning for Ranking Short Social Media Posts. Peng Shi, Jinfeng Rao, Jimmy Lin. NAACL '19. [pdf]

[33] Document Expansion by Query Prediction. Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. arXiv 2019. [pdf]

[34] From Neural Re-Ranking to Neural Ranking: Learning a Sparse Representation for Inverted Indexing. Hamed Zamani. Mostafa Dehghani. W. Bruce Croft. Erik Learned-Miller. Jaap Kamps. CIKM 2018. [pdf]

[35] Explicit Semantic Ranking for Academic Search via Knowledge Graph Embedding. Chenyan Xiong, Russell Power, and Jamie Callan. WWW '17. [pdf]

[36] Word-Entity Duet Representations for Document Ranking. Chenyan Xiong, Jamie Callan, and Tie-Yan Liu. SIGIR '17. [pdf]

[37] Entity-Duet Neural Ranking: Understanding the Role of Knowledge Graph Semantics in Neural Information Retrieval. Zhenghao Liu, Chenyan Xiong, Maosong Sun, and Zhiyuan Liu. ACL '18. [pdf]

[38] Leveraging Entities in Document Retrieval. Krisztian Balog. In: Leveraging Entities in Document Retrieval. Springer 2018. [pdf]

[39] Entity Query Feature Expansion using Knowledge Base Links. Jeffrey Dalton, Laura Dietz, and James Allan. SIGIR '14. [pdf]

[40] Search Result Diversification. Rodrygo L. T. Santos, Craig Macdonald, and Iadh Ounis. Foundations and Trends in Information Retrieval. [pdf]

[41] Learning a Deep Listwise Context Model for Ranking Refinement. Qingyao Ai ,Keping Bi, Jiafeng Guo, W. Bruce Croft. [pdf]

[42] Canjia Li, Yingfei Sun, Ben He, Le Wang, Kai Hui, Andrew Yates, Le Sun, Jungang Xu. NPRF: A Neural Pseudo Relevance Feedback Framework for Ad-hoc Information Retrieval. [pdf]

[43] Pseudo Test Collections for Learning Web Search Ranking Functions. Nima Asadi, Donald Metzler, Tamer Elsayed, Jimmy Lin. SIGIR '11. [pdf]

[44] Neural Ranking Models with Weak Supervision. Mostafa Dehghani, Hamed Zamani, Aliaksei Severyn, Jaap Kamps, W. Bruce Croft. SIGIR '17. [pdf]

[45] Content-Based Weak Supervision for Ad-Hoc Re-Ranking. Sean MacAvaney, Andrew Yates, Kai Hui, Ophir Frieder. [pdf]

[46] Passage Ranking with Weak Supervision. Peng Xu, Xiaofei Ma, Ramesh Nallapati, Bing Xiang. Learning with Limited Labeled Data Workshop at ICLR '19. [pdf]

[47] Cross Domain Regularization for Neural Ranking Models using Adversarial Learning. Daniel Cohen, Bhaskar Mitra, Katja Hofmann, W. Bruce Croft. SIGIR '18. [pdf]

[48] An Axiomatic Approach to Regularizing Neural Ranking Models. Corby Rosset, Bhaskar Mitra, Chenyan Xiong, Nick Craswell, Xia Song, Saurabh Tiwary. SIGIR '19. [pdf]

[49] On the Additivity of Effectiveness Improvements from Neural Ranking Models. Wei Yang, Kuang Lu, Peilin Yang and Jimmy Lin. SIGIR '19. [pdf]

This advanced course consists of 2 hours of lecture per week, four writing assignments that require students to critically read and contrast several scientific articles, and an oral exam. Given the assignments' focus on reading and analyzing scientific literature to achieve a deep understanding of a topic, there will be no weekly tutorials.

Students' final grades will be determined based on the final exam, which will be an oral exam covering material from lectures and their associated readings. In order to be eligible to take the final exam, students must pass all four assignments. Final exam time slots for each student will be announced via email. Students should inform the lecturer of any potential conflicts with the (re-)exam dates as soon as possible.

Assignments will involve reading several scientific papers in order to answer an essay prompt by critically discussing the papers. For each assignment, students will individually read one or more research papers and submit a report discussing the reading and answering the assignment questions. Reports must critically discuss the assigned papers and demonstrate understanding of the topic; simply summarizing them will not be sufficient to receive a passing grade. Reports must cite all sources used. The recommended report length is three pages. Reports are due at noon on the deadline.

Assignments will be given one of four grades: Fail, Pass, Good, or Excellent. Students are allowed to re-submit one failed assignment within two weeks of the assignment deadline. Any assignment that is not submitted by the deadline will be considered failed. Receiving a grade of Excellent gives you one bonus point. Two Good grades count as one Excellent. Each bonus point will improve your final grade by 1/3rd of a point, up to 1 point maximum, on the condition that you pass the final exam. For example, if your final exam grade is 1.6 and you have one bonus point, your final grade will be 1.3. Students who fail the final exam also fail the course regardless of their assignment grades.