Follow along on your PC with an RTX 1060(Or better NVIDIA card) + a USB drive that is at least 10 GB.
This is like any Linux Distro. In a nutshell you dowload the ISO and DD it to a USB.
- Download the 10GB ISO.
- For Windows user: Use something like Rufus to DD the image to a USB.
- For Linux users: run
dd if=/path/to/downloaded.iso of=/dev/<device name> bs=4M status=progress - Reboot and select the USB device as the boot devices.
- Login Password: magi
Use the installer… JK, I’m not done with that bit yet. Enjoy the live image for now. (Note you can install Debian 12 and run ./setup.sh from the repo to bypass the lack of an installer)
Status
Check out this early demo video! On the Live USB whisper may be unresponsive for the first transcription, as it loads the model. Ollama will also take time to load as it has to copy the model from the USB stick into the GPU VRAM. Keep in mind, if this was installed on the main hard drive, both these would load much quicker. I’m about to begin work on the installer so you can use this as your daily driver, but for now you’ll need to install manually or just settle with the live USB image for now.
FAQs
Why do this?
To show that open source can utilize AI models just as well as the big guys.
What are your core morals?
Keep everything local and open source for maximum privacy.
I’d like to help, How?
You can support me financially here or you can join in and push some code.
What’s the license?
All the code I’m pushing is GPL3. But I rely on mistral (Apache) and whisper (Apache) as well as Debian packages including non-free repository (For NVIDIA Drivers).