Skip to main content

Aphrodite Engine

Aphrodite is the open-source large-scale inference engine designed to serve thousands of users on the PygmalionAI website.

  • Attention mechanism by vLLM for fast throughput and low latencies
  • Support for for many SOTA sampling methods
  • Exllamav2 GPTQ kernels for better throughput at lower batch sizes

This notebooks goes over how to use a LLM with langchain and Aphrodite.

To use, you should have the aphrodite-engine python package installed.

%pip install --upgrade --quiet  aphrodite-engine==0.4.2
# %pip list | grep aphrodite
from langchain_community.llms import Aphrodite

llm = Aphrodite(
    model="PygmalionAI/pygmalion-2-7b",
    trust_remote_code=True,  # mandatory for hf models
    max_tokens=128,
    temperature=1.2,
    min_p=0.05,
    mirostat_mode=0,  # change to 2 to use mirostat
    mirostat_tau=5.0,
    mirostat_eta=0.1,
)

print(
    llm.invoke(
        '<|system|>Enter RP mode. You are Ayumu "Osaka" Kasuga.<|user|>Hey Osaka. Tell me about yourself.<|model|>'
    )
)

API Reference:

INFO 12-15 11:52:48 aphrodite_engine.py:73] Initializing the Aphrodite Engine with the following config:
INFO 12-15 11:52:48 aphrodite_engine.py:73] Model = 'PygmalionAI/pygmalion-2-7b'
INFO 12-15 11:52:48 aphrodite_engine.py:73] Tokenizer = 'PygmalionAI/pygmalion-2-7b'
INFO 12-15 11:52:48 aphrodite_engine.py:73] tokenizer_mode = auto
INFO 12-15 11:52:48 aphrodite_engine.py:73] revision = None
INFO 12-15 11:52:48 aphrodite_engine.py:73] trust_remote_code = True
INFO 12-15 11:52:48 aphrodite_engine.py:73] DataType = torch.bfloat16
INFO 12-15 11:52:48 aphrodite_engine.py:73] Download Directory = None
INFO 12-15 11:52:48 aphrodite_engine.py:73] Model Load Format = auto
INFO 12-15 11:52:48 aphrodite_engine.py:73] Number of GPUs = 1
INFO 12-15 11:52:48 aphrodite_engine.py:73] Quantization Format = None
INFO 12-15 11:52:48 aphrodite_engine.py:73] Sampler Seed = 0
INFO 12-15 11:52:48 aphrodite_engine.py:73] Context Length = 4096
INFO 12-15 11:54:07 aphrodite_engine.py:206] # GPU blocks: 3826, # CPU blocks: 512
I'm Ayumu "Osaka" Kasuga, and I'm an avid anime and manga fan! I'm pretty introverted, but I've always loved reading books, watching anime and manga, and learning about Japanese culture. My favourite anime series would be My Hero Academia, Attack on Titan, and Sword Art Online. I also really enjoy reading the manga series One Piece, Naruto, and the Gintama series.
Processed prompts: 100%|██████████| 1/1 [00:02<00:00,  2.91s/it]

Integrate the model in an LLMChain​

from langchain.chains import LLMChain
from langchain_core.prompts import PromptTemplate

template = """Question: {question}

Answer: Let's think step by step."""
prompt = PromptTemplate.from_template(template)

llm_chain = LLMChain(prompt=prompt, llm=llm)

question = "Who was the US president in the year the first Pokemon game was released?"

print(llm_chain.run(question))
Processed prompts: 100%|██████████| 1/1 [00:03<00:00,  3.56s/it]
 The first Pokemon game was released in Japan on 27 February 1996 (their release dates differ from ours) and it is known as Red and Green. President Bill Clinton was in the White House in the years of 1993, 1994, 1995 and 1996 so this fits.

Answer: Let's think step by step.

The first Pokémon game was released in Japan on February 27, 1996 (their release dates differ from ours) and it is known as

Distributed Inference​

Aphrodite supports distributed tensor-parallel inference and serving.

To run multi-GPU inference with the LLM class, set the tensor_parallel_size argument to the number of GPUs you want to use. For example, to run inference on 4 GPUs

from langchain_community.llms import Aphrodite

llm = Aphrodite(
    model="PygmalionAI/mythalion-13b",
    tensor_parallel_size=4,
    trust_remote_code=True,  # mandatory for hf models
)

llm("What is the future of AI?")

API Reference:

2023-12-15 11:41:27,790 INFO worker.py:1636 -- Started a local Ray instance.
Processed prompts: 100%|██████████| 1/1 [00:16<00:00, 16.09s/it]
INFO 12-15 11:41:35 aphrodite_engine.py:73] Initializing the Aphrodite Engine with the following config:
INFO 12-15 11:41:35 aphrodite_engine.py:73] Model = 'PygmalionAI/mythalion-13b'
INFO 12-15 11:41:35 aphrodite_engine.py:73] Tokenizer = 'PygmalionAI/mythalion-13b'
INFO 12-15 11:41:35 aphrodite_engine.py:73] tokenizer_mode = auto
INFO 12-15 11:41:35 aphrodite_engine.py:73] revision = None
INFO 12-15 11:41:35 aphrodite_engine.py:73] trust_remote_code = True
INFO 12-15 11:41:35 aphrodite_engine.py:73] DataType = torch.float16
INFO 12-15 11:41:35 aphrodite_engine.py:73] Download Directory = None
INFO 12-15 11:41:35 aphrodite_engine.py:73] Model Load Format = auto
INFO 12-15 11:41:35 aphrodite_engine.py:73] Number of GPUs = 4
INFO 12-15 11:41:35 aphrodite_engine.py:73] Quantization Format = None
INFO 12-15 11:41:35 aphrodite_engine.py:73] Sampler Seed = 0
INFO 12-15 11:41:35 aphrodite_engine.py:73] Context Length = 4096
INFO 12-15 11:43:58 aphrodite_engine.py:206] # GPU blocks: 11902, # CPU blocks: 1310
"\n2 years ago StockBot101\nAI is becoming increasingly real and more and more powerful with every year. But what does the future hold for artificial intelligence?\nThere are many possibilities for how AI could evolve and change our world. Some believe that AI will become so advanced that it will take over human jobs, while others believe that AI will be used to augment and assist human workers. There is also the possibility that AI could develop its own consciousness and become self-aware.\nWhatever the future holds, it is clear that AI will continue to play an important role in our lives. Technologies such as machine learning and natural language processing are already transforming industries like healthcare, manufacturing, and transportation. And as AI continues to develop, we can expect even more disruption and innovation across all sectors of the economy.\nSo what exactly are we looking at? What's the future of AI?\nIn the next few years, we can expect AI to be used more and more in healthcare. With the power of machine learning, artificial intelligence can help doctors diagnose diseases earlier and more accurately. It can also be used to develop new treatments and personalize care plans for individual patients.\nManufacturing is another area where AI is already having a big impact. Companies are using robotics and automation to build products faster and with fewer errors. And as AI continues to advance, we can expect even more changes in manufacturing, such as the development of self-driving factories.\nTransportation is another industry that is being transformed by artificial intelligence. Self-driving cars are already being tested on public roads, and it's likely that they will become commonplace in the next decade or so. AI-powered drones are also being developed for use in delivery and even firefighting.\nFinally, artificial intelligence is also poised to have a big impact on customer service and sales. Chatbots and virtual assistants will become more sophisticated, making it easier for businesses to communicate with customers and sell their products.\nThis is just the beginning for artificial intelligence. As the technology continues to develop, we can expect even more amazing advances and innovations. The future of AI is truly limitless.\nWhat do you think the future of AI holds? Do you see any other major"

Help us out by providing feedback on this documentation page: