Skip to the content.

About the Book

Get the book here!

The Book

Large language models (LLMs) have recently led to stunning advances in areas like text processing and code generation. The latest generation of language models, including OpenAI’s GPT-4 Omni model, is multimodal. This means they process various data types such as text, images, and audio data. This book teaches readers how to leverage LLMs to perform sophisticated, multimodal data analysis with just a few lines of Python code.

Topics covered in the book include:

The Author

Immanuel Trummer is an associate professor at Cornell University and a member of the Cornell Database Group. His research publications have been selected for various awards, including “Best of SIGMOD”, “Best of VLDB”, “SIGMOD Research Highlight Award”, and “CACM Research Highlight Award”, and often focus on applying LLMs to data problems. He has given various tutorials on LLMs and his courses on data management have received over a million views on YouTube.

Follow the author on X/Twitter!

Book Resources

Chapter 1: Analyzing Data with Large Language Models

Resource Description
Prompt Example prompt for product review classification
Template Prompt template for product review classification

Chapter 2: A Chat with ChatGPT

Resource Description
ChatGPT Link to the ChatGPT Web interface by OpenAI
Review Example review of BananaBook laptop
Review Table Table with classification results for each review
BananaDB Notebook supporting queries on the BananaDB database

Chapter 3: The OpenAI API

Resource Description
Listing 1 Listing available GPT models
Listing 2 Making GPT tell us a story (text completion)
Listing 3 Customizing story generation via parameters

Chapter 4: Analyzing Text Data

Resource Description
Reviews.csv Small set of movie reviews for sentiment classification
Listing 1 Classifying text by underlying sentiment
Biographies.csv Small collection of biographies for text extraction
Listing 2 Extracting structured data from text documents
Textmix.csv Mix of poems and emails, to be used for document clustering
Listing 3 Clustering text documents via their embedding vectors

Chapter 5: Analyzing Structured Data

Resource Description
Games Tabular data set describing video game sales
Games SQLite SQLite database file containing games data
Games NLQI A natural language query interface on the games database
Listing 1 Translating text questions about video games to SQL queries
Listing 2 Translating questions to SQL queries on arbitrary tabular data
Listing 3 Translating questions about movies into Cypher queries

Chapter 6: Analyzing Images and Videos

Resource Description
Listing 1 Answering questions about images
Fruit 1 A JPEG image of an apple
Listing 2 Tagging people in images
Tagging Pictures of people for image tagging
Listing 3 Generating suitable titles for videos
Cars A video of cars on a road

Chapter 7: Analyzing Audio Data

Resource Description
Listing 1 Transcribing audio recordings to text
Audio A sample audio recording for transcription
Listing 2 A voice query interface for tabular data
Listing 3 A speech-to-speech translator

Chapter 8: GPT Alternatives

Resource Description
AI21 - Generic Using generic AI21 models for text completion
AI21 - Paraphrase Using specialized AI21 models for paraphrasing
Anthropic Use Anthropic’s Claude for text completion
Cohere Using Cohere’s Coral model for question answering
Google Using Google’s Gemini model for question answering
Hugging Face Using Hugging Face’s models for sentiment classification

Chapter 9: Optimizing Cost and Quality

Resource Description
Untuned Classifier Basic version of text classification tool
Reviews Training Data set with classified reviews for few-shot learning
Tunable Classifier Tunable version of text classification tool
Prepare Fine-Tuning Prepares model fine-tuning for sentiment classification
Start Fine-Tuning Start fine-tuning for sentiment classification
Check Status Check status of fine-tuning job

Chapter 10: Advanced Frameworks

Resource Description
Listing 1 Using the LangChain framework for text classification
Listing 2 Building an agent for data analysis using the LangChain framework
Listing 3 Enhancing the data analysis agent with a custom currency converter tool
Listing 4 Building a question-answering tool over multimodal data with LlamaIndex
Banana Reports Collection of business reports in different file formats

Video Tutorial

Tutorial Resources

Resource Description
Text Classification Classifying reviews based on the underlying sentiment
Image Analysis Answering questions about pairs of images
Speech Transcription Transcribing audio data (speech) to text representation
Text-to-SQL Querying an SQLite database in natural language
Reviews.csv Small set of movie reviews for sentiment classification
games.db SQLite database with data on vide game sales