Course Page - IIT Madras Degree Program

Degree Level Course

Introduction to Natural Language Processing (i-NLP)

Natural language (NL) refers to the language spoken/written by humans. NL is the primary mode
of communication for humans. With the growth of the world wide web, data in the form of text
has grown exponentially. It calls for the development of algorithms and techniques for
processing natural language for the automation and development of intelligent machines: Natural
Language Processing (NLP).

On the completing the course, the participant will learn the following:
1. Why is processing language computationally hard and why specialized techniques need
to be developed to process texts?
2. Knowledge and in-depth understanding of linguistics techniques and classical (statistical)
approaches (pre-deep learning era) to NLP and their limitations.
3. Knowledge and in-depth understanding of deep learning approaches (RNN and CNN) to
NLP.
4. Knowledge and in-depth understanding of Attention Mechanism, Transformers and Large
Language Models (LLMs)
5. Ability to read and understand latest NLP-related research papers.
6. Ability to identify applicable NLP technique to solve a real-world problem involving text
processing.
7. Ability to implement NLP models and algorithms for problems related to text processing.
8. Ability to develop applications based on textual generative models (LLMs - Large
Language Models)

by Parameswari Krishnamurthy , Rahul Mishra

Course ID: BSDA5005

Course Credits: 4

Course Type: Elective

Pre-requisites: BSCS3004 - Deep Learning

About the Instructors

Parameswari Krishnamurthy

Assistant Professor, Language Technologies Research Center (LTRC), IIIT Hyderabad

Parameswari Krishnamurthy is as an Assistant Professor at the Language Technologies Research Centre, International Institute of Information Technology, Hyderabad. Her doctoral research at the University of Hyderabad delved into Computational Linguistics, with a focus on Telugu-Tamil Machine Translation, emphasizing divergence. Prior to her current position, she held the role of Assistant Professor at the University of Hyderabad. Krishnamurthy's expertise lies in Computational Linguistics and Machine Translation, spanning areas such as Morphological Analysis & Generation, Parsing, Linguistic Divergence, Contrastive Grammar, and Corpus Linguistics. She actively participates in numerous Natural Language Processing projects, particularly those concerning Indian languages and Dravidian languages, aiming to advance core technology tools and NLP applications.

Rahul Mishra

Assistant Professor, Language Technologies Research Center (LTRC), Computer Science, IIIT Hyderabad

Rahul Mishra is an Assistant Professor at IIIT Hyderabad's Language Technology Research Centre (LTRC), where his research focuses on Controllable Text Summarization, Misinformation Detection, Model Explainability, Graph Representation Learning, and Natural Language Generation. Previously, he served as a senior postdoctoral researcher at the University of Geneva, Switzerland, specializing in biomedical NLP. Prior to that, as a Senior Staff Engineer/Researcher, he contributed to research projects at Samsung Research Lab in Bangalore, optimizing and benchmarking large language models on process in memory (PIM) enabled GPUs. He holds a PhD from the University of Stavanger, Norway and an M.Tech from IIIT Delhi. During his doctoral studies, he also worked as a visiting researcher at the Computer Science Department of ETH, Zurich, Switzerland and University of Hannover, Germany. Before pursuing his PhD, he worked as an NLP data scientist in automatic vehicle diagnostic department at KPIT Technologies, Pune, focusing on automatic fact extraction from car service manuals. Prior to that, he also held roles as a consultant researcher at Tata Research Development and Design Centre (TRDDC) and a research intern at IBM Research Bangalore.

WEEK 1	Introduction to Natural Language (NL) Why is it hard to process a natural language? Levels of Language Processing, Linguistic Fundamentals for NLP
WEEK 2	Text Processing and Preprocessing: Tokenization, Normalization, Stop word removal, Stemming, lemmatization, Morphological Analysis & Finite State Transducers (Code demo: Tokenization, Stop-Word-Removel, Stemming, Lemmatization)
WEEK 3	Part-of-speech tagging and Named Entities; Sequence Models: Hidden-Markov Models, MEMM and CRF; Classification Models: Naïve Bayes, Logistic Regression, Clustering. (Code demo: HMM, CRF, Naive Bayes, LR, KNN)
WEEK 4	Syntax and Parsing: Constituency parsing, Dependency parsing, Parsing algorithms; Meaning Representation: Logical Semantics, Semantic Role Labelling (Code demo: Transition-based Parser, Graph-based parser, SRL-CRF (Prop bank.))
WEEK 5	Distributional Semantics, n-gram and Word2Vec, GloVe; Discourse Processing: Anaphora and Coreference Resolution and Discourse Connectives. Machine Translation (Code demo: Word2Vec, Glove, openNMT)
WEEK 6	Recurrent neural networks, LSTMs/GRUs, Neural Sequence Models, Contextualized Word Embeddings: TagLM, ELMO, ULMFIT, etc., Attention Mechanism (Code demo: LSTM/GRU)
WEEK 7	Transformers, Self-attention Mechanism, Sub-word tokenization, Positional encoding, Pre-trained Language Models (PLMs): BERT, GPT, etc. Fine-tuning and transfer learning (Code demo: Transformers).
WEEK 8	Large Language Models (LLMs) Parameter Efficient Fine Tuning: Prefix-coding, LORA, QLORA, etc. Emergent Behavior: In-context learning, Instruction Fine Tuning, RLHF, DPO (Code demo: LLMs with Hugging face)
WEEK 9	Natural Language Generation, Decoding schemes: greedy, Random sampling, Top-k, Top-p, Speculative sampling, etc.
WEEK 10	NLP applications: QnA, Summarization, NLI, Fact-checking, etc. Retrieval Augmented generation (RAG). (Code demo: Summarisation)
WEEK 11	Model Explainability: Attention maps, Attention-flow/rollout, Integrated gradients, etc. (Code demo: Attension visualization and Integrated gradients )
WEEK 12	Ethical and cultural considerations and biases.