About the Book
Get the book here!
The Book
Large language models (LLMs) have recently led to stunning advances in areas like text processing and code generation. The latest generation of language models, including OpenAI’s GPT-4 Omni model, is multimodal. This means they process various data types such as text, images, and audio data. This book teaches readers how to leverage LLMs to perform sophisticated, multimodal data analysis with just a few lines of Python code.
Topics covered in the book include:
- Using and configuring LLMs via OpenAI’s Python API
- Analyzing text data with LLMs
- Analyzing images and videos
- Natural language query interfaces over tabular data
- Prompt engineering and few-shot learning
- Optimizing cost and quality by fine-tuning
- LLM providers (OpenAI, Anthropic, Cohere, Google, HuggingFace, AI21)
- Building complex data analysis pipelines with LangChain and LlamaIndex
- Building LLM-based agents for data analysis
- Foundations of Transformer networks and transfer learning
The Author
Immanuel Trummer is an associate professor at Cornell University and a member of the Cornell Database Group. His research publications have been selected for various awards, including “Best of SIGMOD”, “Best of VLDB”, “SIGMOD Research Highlight Award”, and “CACM Research Highlight Award”, and often focus on applying LLMs to data problems. He has given various tutorials on LLMs and his courses on data management have received over a million views on YouTube.
Follow the author on X/Twitter!
Book Resources
Chapter 1: Analyzing Data with Large Language Models
| Resource |
Description |
| Prompt |
Example prompt for product review classification |
| Template |
Prompt template for product review classification |
Chapter 2: A Chat with ChatGPT
| Resource |
Description |
| ChatGPT |
Link to the ChatGPT Web interface by OpenAI |
| Review |
Example review of BananaBook laptop |
| BananaDB |
Notebook supporting queries on the BananaDB database |
Chapter 3: The OpenAI API
| Resource |
Description |
| Listing 1 |
Listing available GPT models |
| Listing 2 |
Making GPT tell us a story (text completion) |
| Listing 3 |
Customizing story generation via parameters |
Chapter 4: Analyzing Text Data
| Resource |
Description |
| Reviews.csv |
Small set of movie reviews for sentiment classification |
| Listing 1 |
Classifying text by underlying sentiment |
| Biographies.csv |
Small collection of biographies for text extraction |
| Listing 2 |
Extracting structured data from text documents |
| Textmix.csv |
Mix of poems and emails, to be used for document clustering |
| Listing 3 |
Clustering text documents via their embedding vectors |
Chapter 5: Analyzing Structured Data
| Resource |
Description |
| Games |
Tabular data set describing video game sales |
| Games SQLite |
SQLite database file containing games data |
| Games NLQI |
A natural language query interface on the games database |
| Listing 1 |
Translating text questions about video games to SQL queries |
| Listing 2 |
Example prompt for translating question to SQL query |
| Listing 3 |
Translating questions to SQL queries on arbitrary tabular data |
| Listing 4 |
Example interaction with natural language query interface |
| Listing 5 |
Example prompt for translating questions into Cypher queries |
| Listing 6 |
Translating questions about movies into Cypher queries |
Chapter 6: Analyzing Images and Videos
| Resource |
Description |
| Listing 1 |
Answering questions about images |
| Fruit 1 |
A JPEG image of an apple |
| Listing 2 |
Tagging people in images |
| Tagging |
Pictures of people for image tagging |
| Listing 3 |
Generating suitable titles for videos |
| Cars |
A video of cars on a road |
Chapter 7: Analyzing Audio Data
| Resource |
Description |
| Listing 1 |
Transcribing audio recordings to text |
| Audio |
A sample audio recording for transcription |
| Listing 2 |
A voice query interface for tabular data |
| Listing 3 |
A speech-to-speech translator |
Chapter 8: GPT Alternatives
| Resource |
Description |
| AI21 - Generic |
Using generic AI21 models for text completion |
| AI21 - Paraphrase |
Using specialized AI21 models for paraphrasing |
| Anthropic |
Use Anthropic’s Claude for text completion |
| Cohere |
Using Cohere’s Coral model for question answering |
| Google |
Using Google’s Gemini model for question answering |
| Hugging Face |
Using Hugging Face’s models for sentiment classification |
Chapter 9: Optimizing Cost and Quality
Chapter 10: Advanced Frameworks
| Resource |
Description |
| Listing 1 |
Using the LangChain framework for text classification |
| Listing 2 |
Building an agent for data analysis using the LangChain framework |
| Listing 3 |
Enhancing the data analysis agent with a custom currency converter tool |
| Listing 4 |
Building a question answering tool over multimodal data with LlamaIndex |
| Resource |
Description |
| Visualizations |
Notebook for visualizing attention in the BERT model |
Chapter 12: Transfer Learning
| Resource |
Description |
| Roberta XLM |
Example description of Roberta Transformer model |
Video Tutorial
Tutorial Resources
