llm-recipes
NotebookLM - Self-hosted Audiobook Generation Platform
Overview
NotebookLM is a self-hosted platform designed to create, manage, and enhance audiobooks. This platform leverages various tools and models to parse, transcribe, and improve manuscripts, ultimately providing high-quality audio content.
- Project Dhwani
- [Source] (https://github.com/slabstech/dhwani)
- Roadmap: Product Launch Roadmap Read more
Steps
- python audiobook.py
- converts script pdf into structured scene json using mistral-large API.
- triggers Text to Speech server for speech/dialogs and store audio files in resources folder
- triggers Text to Sound server for background sound/music and store audio files in resources folder
- Combines generated audio from speech and sound modules and creates audibook for the provided script
- Prompts for Audiobook: Creating a storyline using prompts Read more
Current Status
| Module | Description | Tech | Source | Status |
|---|---|---|---|---|
| Script parser | Convert to scene structured json | LLM Multi-shot Prompt | script_parser.py | Done |
| TTS server | Parler-tts server to generate speech for dialogs in scenes | Parler-tts | tts_generator.py | Done |
| AudioGen module | Audiocraft/magnet background sound creator | Audiocraft/magnet | In progress | |
| Basic audiobook | Linear workflow for full audio creation without scene logic | python | audiobook.py | In progress |
Documentation
- Challenge: project challenge. Read more
- Approach: Approach in solving the problem. Read more
- Workflow: Worflow for audiobook generation. Read more
- Setup: Follow the setup guide to build an End-to-End platform for Audibook Generation/Notebook LLM.
- UX: UX design for Audiobook generation Read more
- Database Schema: Schema for Scene Editing and Media content Read more
An experiment to build a production grade audiobook content generator system to help publishers build on their IP and reach larger audience.
Inference Speed
| GPU Model | torch.compile |
batch generation + chunking | Speech Generator (secs) | Script Parser (Local) (secs) | Script Parser (Online) (secs) | Total Time (secs) |
|---|---|---|---|---|---|---|
| GTX 1060 | no | no | 2775.97 | - | 87.37 | 2863.34 |
| GTX 1060 | no | yes | 663.81 | - | 88.96 | 683.81 |
| RTX 4050 | no | no | 489.93 | - | 93.82 | 583.75 |
| RTX 4050 | no | yes | - | - | - | |
| RTX 4050 | yes | – | - | – | – |
Reference
-
NotebookLlama: Additional resources and use cases. Learn more
- TTS - Generation Docker - https://github.com/ashleykleynhans/tts-generation-docker
- https://github.com/rsxdalv/tts-generation-webui
-
WishList - RTX 4090 GPU based computer for building End to End product inhouse. Buy Me Razer Blade 18 or ROG Zephyrus M16
-
Specific Speakers - Jon, Lea, Gary, Jenna, Mike, Laura
- Inference Tips