rustformers/bloom-ggml

GGML converted versions of BigScience's Bloom models

Description

BLOOM is an autoregressive Large Language Model (LLM), trained to continue text from a prompt on vast amounts of text data using industrial-scale computational resources. As such, it is able to output coherent text in 46 languages and 13 programming languages that is hardly distinguishable from text written by humans. BLOOM can also be instructed to perform text tasks it hasn't been explicitly trained for, by casting them as text generation tasks.

Converted Models

Name	Based on	Type	Container	GGML Version
bloom-1b1-f16.bin	bigscience/bloom-1b1	F16	GGML	V3
bloom-1b1-q4_0.bin	bigscience/bloom-1b1	Q4_0	GGML	V3
bloom-1b1-q4_0-ggjt.bin	bigscience/bloom-1b1	Q4_0	GGJT	V3
bloom-1b1-q5_1.bin	bigscience/bloom-1b1	Q5_1	GGML	V3
bloom-1b1-q5_1-ggjt.bin	bigscience/bloom-1b1	Q5_1	GGJT	V3
bloom-1b7-f16.bin	bigscience/bloom-1b7	F16	GGML	V3
bloom-1b7-q4_0.bin	bigscience/bloom-1b7	Q4_0	GGML	V3
bloom-1b7-q4_0-ggjt.bin	bigscience/bloom-1b7	Q4_0	GGJT	V3
bloom-1b7-q5_1.bin	bigscience/bloom-1b7	Q5_1	GGML	V3
bloom-1b7-q5_1-ggjt.bin	bigscience/bloom-1b7	Q5_1	GGJT	V3
bloom-3b-f16.bin	bigscience/bloom-3b	F16	GGML	V3
bloom-3b-q4_0.bin	bigscience/bloom-3b	Q4_0	GGML	V3
bloom-3b-q4_0-ggjt.bin	bigscience/bloom-3b	Q4_0	GGJT	V3
bloom-3b-q5_1.bin	bigscience/bloom-3b	Q5_1	GGML	V3
bloom-3b-q5_1-ggjt.bin	bigscience/bloom-3b	Q5_1	GGJT	V3
bloom-560m-f16.bin	bigscience/bloom-560m	F16	GGML	V3
bloom-560m-q4_0.bin	bigscience/bloom-560m	Q4_0	GGML	V3
bloom-560m-q4_0-ggjt.bin	bigscience/bloom-560m	Q4_0	GGJT	V3
bloom-560m-q5_1.bin	bigscience/bloom-560m	Q5_1	GGML	V3
bloom-560m-q5_1-ggjt.bin	bigscience/bloom-560m	Q5_1	GGJT	V3

Usage

Python via llm-rs:

Installation

Via pip: pip install llm-rs

Run inference

from llm_rs import AutoModel

#Load the model, define any model you like from the list above as the `model_file`
model = AutoModel.from_pretrained("rustformers/bloom-ggml",model_file="bloom-3b-q4_0-ggjt.bin")

#Generate
print(model.generate("The meaning of life is"))

Rust via Rustformers/llm:

Installation

git clone --recurse-submodules https://github.com/rustformers/llm.git
cd llm
cargo build --release

Run inference

cargo run --release -- bloom infer -m path/to/model.bin  -p "Tell me how cool the Rust programming language is:"