About the Book
Large language models (LLMs) have recently led to stunning advances in areas like text processing and code generation. The latest generation of language models, including OpenAI’s GPT-4 Omni model, is multimodal. This means they process various data types such as text, images, and audio data. This book teaches readers how to leverage LLMs to perform sophisticated, multimodal data analysis with just a few lines of Python code.
Topics covered in the book include:
- Using LLMs via OpenAI’s Python library
- Analyzing text data with LLMs
- Analyzing images and videos
- Natural language interfaces for tabular data
- Prompt engineering and few-shot learning
- Optimizing cost and quality by fine-tuning
- Anthropic, Cohere, Google, HuggingFace, and AI21
- Building applications with LangChain and LlamaIndex
- Building LLM-based agents for data analysis
About the Author
Immanuel Trummer is an associate professor at Cornell University and a member of the Cornell Database Group. His research publications have been selected for various awards, including “Best of SIGMOD”, “Best of VLDB”, “SIGMOD Research Highlight Award”, and “CACM Research Highlight Award”, and often focus on applying LLMs to data problems. He has given various tutorials on LLMs and his courses on data management have received over a million views on YouTube.
Follow the author on X/Twitter!
Book Resources
Chapter 1: Analyzing Data with Large Language Models
| Resource |
Description |
| Prompt |
Example prompt for product review classification |
| Template |
Prompt template for product review classification |
Chapter 2: A Chat with ChatGPT
| Resource |
Description |
| ChatGPT |
Link to the ChatGPT Web interface by OpenAI |
| Review |
Example review of BananaBook laptop |
| Review Table |
Table with classification results for each review |
| BananaDB |
Notebook supporting queries on the BananaDB database |
Chapter 3: The OpenAI Python Library
| Resource |
Description |
| Listing 1 |
Listing available GPT models |
| Listing 2 |
Making GPT tell us a story (text completion) |
| Listing 3 |
Customizing story generation via parameters |
Chapter 4: Analyzing Text Data
| Resource |
Description |
| Reviews.csv |
Small set of movie reviews for sentiment classification |
| Listing 1 |
Classifying text by underlying sentiment |
| Biographies.csv |
Small collection of biographies for text extraction |
| Listing 2 |
Extracting structured data from text documents |
| Textmix.csv |
Mix of poems and emails, to be used for document clustering |
| Listing 3 |
Clustering text documents via their embedding vectors |
Chapter 5: Analyzing Structured Data
| Resource |
Description |
| Games |
Tabular data set describing video game sales |
| Games SQLite |
SQLite database file containing games data |
| Games NLQI |
A natural language query interface on the games database |
| Listing 1 |
Translating text questions about video games to SQL queries |
| Listing 2 |
Translating questions to SQL queries on arbitrary tabular data |
| Listing 3 |
Translating questions about movies into Cypher queries |
Chapter 6: Analyzing Images and Videos
| Resource |
Description |
| Listing 1 |
Answering questions about images |
| Fruit 1 |
A JPEG image of an apple |
| Listing 2 |
Tagging people in images |
| Tagging |
Pictures of people for image tagging |
| Listing 3 |
Generating suitable titles for videos |
| Cars |
A video of cars on a road |
Chapter 7: Analyzing Audio Data
| Resource |
Description |
| Listing 1 |
Transcribing audio recordings to text |
| Audio |
A sample audio recording for transcription |
| Listing 2 |
A voice query interface for tabular data |
| Listing 3 |
A speech-to-speech translator |
Chapter 8: GPT Alternatives
| Resource |
Description |
| AI21 - Generic |
Using generic AI21 models for text completion |
| AI21 - Paraphrase |
Using specialized AI21 models for paraphrasing |
| Anthropic |
Use Anthropic’s Claude for text completion |
| Cohere |
Using Cohere’s Coral model for question answering |
| Google |
Using Google’s Gemini model for question answering |
| Hugging Face |
Using Hugging Face’s models for sentiment classification |
Chapter 9: Optimizing Cost and Quality
Chapter 10: Software Frameworks
| Resource |
Description |
| Listing 1 |
Using the LangChain framework for text classification |
| Listing 2 |
Building an agent for data analysis using the LangChain framework |
| Listing 3 |
Enhancing the data analysis agent with a custom currency converter tool |
| Listing 4 |
Building a question-answering tool over multimodal data with LlamaIndex |
| Banana Reports |
Collection of business reports in different file formats |
Video Tutorial
Tutorial Resources
