Skip to the content.

About the Book

Get the book here!

The Book

Large language models (LLMs) have recently led to stunning advances in areas like text processing and code generation. The latest generation of language models, including OpenAI’s GPT-4 Omni model, is multimodal. This means they process various data types such as text, images, and audio data. This book teaches readers how to leverage LLMs to perform sophisticated, multimodal data analysis with just a few lines of Python code.

Topics covered in the book include:

The Author

Immanuel Trummer is an associate professor at Cornell University and a member of the Cornell Database Group. His research publications have been selected for various awards, including “Best of SIGMOD”, “Best of VLDB”, “SIGMOD Research Highlight Award”, and “CACM Research Highlight Award”, and often focus on applying LLMs to data problems. He has given various tutorials on LLMs and his courses on data management have received over a million views on YouTube.

Book Resources

Chapter 1: Analyzing Data with Large Language Models

Resource Description
Prompt Example prompt for product review classification
Template Prompt template for product review classification

Chapter 2: A Chat with ChatGPT

Resource Description
ChatGPT Link to the ChatGPT Web interface by OpenAI
Review Example review of BananaBook laptop
BananaDB Notebook supporting queries on the BananaDB database

Chapter 3: The OpenAI API

Resource Description
Listing 1 Listing available GPT models
Listing 2 Making GPT tell us a story (text completion)
Listing 3 Customizing story generation via parameters

Chapter 4: Analyzing Text Data

Resource Description
Reviews.csv Small set of movie reviews for sentiment classification
Listing 1 Classifying text by underlying sentiment
Biographies.csv Small collection of biographies for text extraction
Listing 2 Extracting structured data from text documents
Textmix.csv Mix of poems and emails, to be used for document clustering
Listing 3 Clustering text documents via their embedding vectors

Chapter 5: Analyzing Structured Data

Resource Description
Games Tabular data set describing video game sales
Games SQLite SQLite database file containing games data
Games NLQI A natural language query interface on the games database
Listing 1 Translating text questions about video games to SQL queries
Listing 2 Example prompt for translating question to SQL query
Listing 3 Translating questions to SQL queries on arbitrary tabular data
Listing 4 Example interaction with natural language query interface
Listing 5 Example prompt for translating questions into Cypher queries
Listing 6 Translating questions about movies into Cypher queries

Chapter 6: Analyzing Images and Videos

Resource Description
Listing 1 Answering questions about images
Fruit 1 A JPEG image of an apple
Listing 2 Tagging people in images
Tagging Pictures of people for image tagging
Listing 3 Generating suitable titles for videos
Cars A video of cars on a road

Chapter 7: Analyzing Audio Data

Resource Description
Listing 1 Transcribing audio recordings to text
Audio A sample audio recording for transcription
Listing 2 A voice query interface for tabular data
Listing 3 A speech-to-speech translator

Chapter 8: GPT Alternatives

Resource Description
AI21 - Generic Using generic AI21 models for text completion
AI21 - Paraphrase Using specialized AI21 models for paraphrasing
Anthropic Use Anthropic’s Claude for text completion
Cohere Using Cohere’s Coral model for question answering
Google Using Google’s Gemini model for question answering
Hugging Face Using Hugging Face’s models for sentiment classification

Chapter 9: Optimizing Cost and Quality

Resource Description
Untuned Classifier Basic version of text classification tool
Tunable Classifier Tunable version of text classification tool
Prepare Fine-Tuning Prepares model fine-tuning for sentiment classification
Start Fine-Tuning Start fine-tuning for sentiment classification
Check Status Check status of fine-tuning job

Chapter 10: Advanced Frameworks

Resource Description
Listing 1 Using the LangChain framework for text classification
Listing 2 Building an agent for data analysis using the LangChain framework
Listing 3 Enhancing the data analysis agent with a custom currency converter tool
Listing 4 Building a question answering tool over multimodal data with LlamaIndex

Chapter 11: The Transformer

Resource Description
Visualizations Notebook for visualizing attention in the BERT model

Chapter 12: Transfer Learning

Resource Description
Roberta XLM Example description of Roberta Transformer model