Pydantic
Pydantic is the most widely used data validation library for Python [1]. Guardrails extends pydantic to provide structured validations over LLM outputs and describe output structure for LLM responses.
Guardrails has first class Pydantic support, and we encourage users to use Pydantic as the primary way to specify guards.
Structured outputs
Guardrails uses pydantic to describe and validate structured output. For unstructured output, see Strings.
Let's say you want an LLM to generate fake pets. We can model a Pet as class that inherits from BaseModel. Each field can take descriptions and validators.
This class can be passed to a guard along with a prompt using the from_pydantic initializer, which takes an output_class parameter. When invoked, the LLM returns output formatted to the pydantic structure. In the prompt, we need to make sure we keep an output formatter suffix - ${gr.complete_json_suffix_v2}. This tells our LLM to respond with json.
from guardrails import Guard
import openai
prompt = """
What kind of pet should I get and what should I name it?
${gr.complete_json_suffix_v2}
"""
guard = Guard.from_pydantic(output_class=Pet, prompt=prompt)
raw_llm_output, validated_output = guard(
llm_api=openai.Completion.create,
engine="text-davinci-003"
)
print(validated_output)
Async event loop found, but guard was invoked synchronously.For validator parallelization, please call `validate_async` instead.
{'pet_type': 'Dog', 'name': 'Max'}
Structured output with validation
Now that we have our LLM responding to us in JSON with the structured information we're asking for, we can add validations and corrective actions. Below, we've added a validator to the 'name' field that ensures the name cannot be null. We've also added an on_fail action of "reask" if the name is null. What this does is reasks the LLM if the validation fails. Check the Validators API Spec for a list of standard validators, or you can write your own.
from guardrails.validators import ValidLength, TwoWords
from rich import print
class Pet(BaseModel):
pet_type: str = Field(description="Species of pet")
name: str = Field(description="a unique pet name", validators=[ValidLength(min=8, max=32, on_fail='reask')])
guard = Guard.from_pydantic(output_class=Pet, prompt=prompt)
raw_llm_output, validated_output = guard(
llm_api=openai.Completion.create,
engine="text-davinci-003",
max_tokens=1024,
temperature=1
)
print(guard.state.most_recent_call.tree)
Async event loop found, but guard was invoked synchronously.For validator parallelization, please call `validate_async` instead.
Async event loop found, but guard was invoked synchronously.For validator parallelization, please call `validate_async` instead.
Logs ├── ╭────────────────────────────────────────────────── Step 0 ───────────────────────────────────────────────────╮ │ │ ╭──────────────────────────────────────────────── Prompt ─────────────────────────────────────────────────╮ │ │ │ │ │ │ │ │ │ What kind of pet should I get and what should I name it? Give me some details about it's breed │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ Given below is XML that describes the information to extract from this document and the tags to extract │ │ │ │ │ it into. │ │ │ │ │ │ │ │ │ │ <output> │ │ │ │ │ <string name="pet_type" description="Species of pet"/> │ │ │ │ │ <string name="name" format="length: min=8 max=32" description="a unique pet name"/> │ │ │ │ │ </output> │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ ONLY return a valid JSON object (no other text is necessary), where the key of the field in JSON is the │ │ │ │ │ `name` attribute of the corresponding XML, and the value is of the type specified by the corresponding │ │ │ │ │ XML's tag. The JSON MUST conform to the XML format, including any types and format requests e.g. │ │ │ │ │ requests for lists, objects and specific types. Be correct and concise. │ │ │ │ │ │ │ │ │ │ Here are examples of simple (XML, JSON) pairs that show the expected behavior: │ │ │ │ │ - `<string name='foo' format='two-words lower-case' />` => `{'foo': 'example one'}` │ │ │ │ │ - `<list name='bar'><string format='upper-case' /></list>` => `{"bar": ['STRING ONE', 'STRING TWO', │ │ │ │ │ etc.]}` │ │ │ │ │ - `<object name='baz'><string name="foo" format="capitalize two-words" /><integer name="index" │ │ │ │ │ format="1-indexed" /></object>` => `{'baz': {'foo': 'Some String', 'index': 1}}` │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ Json Output: │ │ │ │ │ │ │ │ │ │ │ │ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │ │ │ ╭──────────────────────────────────────────── Message History ────────────────────────────────────────────╮ │ │ │ │ ┏━━━━━━┳━━━━━━━━━┓ │ │ │ │ │ ┃ Role ┃ Content ┃ │ │ │ │ │ ┡━━━━━━╇━━━━━━━━━┩ │ │ │ │ │ └──────┴─────────┘ │ │ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │ │ │ ╭──────────────────────────────────────────── Raw LLM Output ─────────────────────────────────────────────╮ │ │ │ │ {"pet_type": "Dog", "name": "Fido"} │ │ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │ │ │ ╭─────────────────────────────────────────── Validated Output ────────────────────────────────────────────╮ │ │ │ │ { │ │ │ │ │ 'pet_type': 'Dog', │ │ │ │ │ 'name': FieldReAsk( │ │ │ │ │ incorrect_value='Fido', │ │ │ │ │ fail_results=[ │ │ │ │ │ FailResult( │ │ │ │ │ outcome='fail', │ │ │ │ │ metadata=None, │ │ │ │ │ error_message='Value has length less than 8. Please return a longer output, that is │ │ │ │ │ shorter than 32 characters.', │ │ │ │ │ fix_value='Fidooooo' │ │ │ │ │ ) │ │ │ │ │ ], │ │ │ │ │ path=['name'] │ │ │ │ │ ) │ │ │ │ │ } │ │ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ └── ╭────────────────────────────────────────────────── Step 1 ───────────────────────────────────────────────────╮ │ ╭──────────────────────────────────────────────── Prompt ─────────────────────────────────────────────────╮ │ │ │ │ │ │ │ I was given the following JSON response, which had problems due to incorrect values. │ │ │ │ │ │ │ │ { │ │ │ │ "pet_type": "Dog", │ │ │ │ "name": { │ │ │ │ "incorrect_value": "Fido", │ │ │ │ "error_messages": [ │ │ │ │ "Value has length less than 8. Please return a longer output, that is shorter than 32 │ │ │ │ characters." │ │ │ │ ] │ │ │ │ } │ │ │ │ } │ │ │ │ │ │ │ │ Help me correct the incorrect values based on the given error messages. │ │ │ │ │ │ │ │ Given below is XML that describes the information to extract from this document and the tags to extract │ │ │ │ it into. │ │ │ │ │ │ │ │ <output> │ │ │ │ <string name="pet_type" description="Species of pet"/> │ │ │ │ <string name="name" format="length: min=8 max=32" description="a unique pet name"/> │ │ │ │ </output> │ │ │ │ │ │ │ │ │ │ │ │ ONLY return a valid JSON object (no other text is necessary), where the key of the field in JSON is the │ │ │ │ `name` attribute of the corresponding XML, and the value is of the type specified by the corresponding │ │ │ │ XML's tag. The JSON MUST conform to the XML format, including any types and format requests e.g. │ │ │ │ requests for lists, objects and specific types. Be correct and concise. If you are unsure anywhere, │ │ │ │ enter `null`. │ │ │ │ │ │ │ │ │ │ │ │ Json Output: │ │ │ │ │ │ │ │ │ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │ │ ╭───────────────────────────────────────────── Instructions ──────────────────────────────────────────────╮ │ │ │ │ │ │ │ You are a helpful assistant only capable of communicating with valid JSON, and no other text. │ │ │ │ │ │ │ │ ONLY return a valid JSON object (no other text is necessary), where the key of the field in JSON is the │ │ │ │ `name` attribute of the corresponding XML, and the value is of the type specified by the corresponding │ │ │ │ XML's tag. The JSON MUST conform to the XML format, including any types and format requests e.g. │ │ │ │ requests for lists, objects and specific types. Be correct and concise. If you are unsure anywhere, │ │ │ │ enter `null`. │ │ │ │ │ │ │ │ Here are examples of simple (XML, JSON) pairs that show the expected behavior: │ │ │ │ - `<string name='foo' format='two-words lower-case' />` => `{'foo': 'example one'}` │ │ │ │ - `<list name='bar'><string format='upper-case' /></list>` => `{"bar": ['STRING ONE', 'STRING TWO', │ │ │ │ etc.]}` │ │ │ │ - `<object name='baz'><string name="foo" format="capitalize two-words" /><integer name="index" │ │ │ │ format="1-indexed" /></object>` => `{'baz': {'foo': 'Some String', 'index': 1}}` │ │ │ │ │ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │ │ ╭──────────────────────────────────────────── Message History ────────────────────────────────────────────╮ │ │ │ No message history. │ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │ │ ╭──────────────────────────────────────────── Raw LLM Output ─────────────────────────────────────────────╮ │ │ │ {"pet_type": "Dog", "name": "FidoFido"} │ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │ │ ╭─────────────────────────────────────────── Validated Output ────────────────────────────────────────────╮ │ │ │ {'pet_type': 'Dog', 'name': 'FidoFido'} │ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Nested structured Output
Finally, we can structure and validate nested pydantic models. For example, if we wanted to ask for additional metadata, we could do something like this
class BreedInformation(BaseModel):
breed_name: str = Field(description="Specific pet breed")
common_colors: str = Field(description="common colors for this pet breed")
life_expectancy: str = Field(description="how long animals of this breed tend to live")
class Pet(BaseModel):
pet_type: str = Field(description="A type of pet")
name: str = Field(description="a unique pet name", validators=[ValidLength(min=8, max=32, on_fail='reask')])
breed: BreedInformation = Field(description="information about the breed of pet")
prompt = """
What kind of pet should I get and what should I name it? Give me some details about it's breed
${gr.complete_json_suffix_v2}
"""
guard = Guard.from_pydantic(output_class=Pet, prompt=prompt)
raw_llm_output, validated_output = guard(
llm_api=openai.Completion.create,
engine="text-davinci-003",
max_tokens=1024,
temperature=1
)
print(guard.state.most_recent_call.tree)
Async event loop found, but guard was invoked synchronously.For validator parallelization, please call `validate_async` instead.
Logs
└── ╭────────────────────────────────────────────────── Step 0 ───────────────────────────────────────────────────╮
│ ╭──────────────────────────────────────────────── Prompt ─────────────────────────────────────────────────╮ │
│ │ │ │
│ │ What kind of pet should I get and what should I name it? Give me some details about it's breed │ │
│ │ │ │
│ │ │ │
│ │ Given below is XML that describes the information to extract from this document and the tags to extract │ │
│ │ it into. │ │
│ │ │ │
│ │ <output> │ │
│ │ <string name="pet_type" description="A type of pet"/> │ │
│ │ <string name="name" format="length: min=8 max=32" description="a unique pet name"/> │ │
│ │ <object name="breed" description="information about the breed of pet"> │ │
│ │ <string name="breed_name" description="Specific pet breed"/> │ │
│ │ <string name="common_colors" description="common colors for this pet breed"/> │ │
│ │ <string name="life_expectancy" description="how long animals of this breed tend to live"/> │ │
│ │ </object> │ │
│ │ </output> │ │
│ │ │ │
│ │ │ │
│ │ ONLY return a valid JSON object (no other text is necessary), where the key of the field in JSON is the │ │
│ │ `name` attribute of the corresponding XML, and the value is of the type specified by the corresponding │ │
│ │ XML's tag. The JSON MUST conform to the XML format, including any types and format requests e.g. │ │
│ │ requests for lists, objects and specific types. Be correct and concise. │ │
│ │ │ │
│ │ Here are examples of simple (XML, JSON) pairs that show the expected behavior: │ │
│ │ - `<string name='foo' format='two-words lower-case' />` => `{'foo': 'example one'}` │ │
│ │ - `<list name='bar'><string format='upper-case' /></list>` => `{"bar": ['STRING ONE', 'STRING TWO', │ │
│ │ etc.]}` │ │
│ │ - `<object name='baz'><string name="foo" format="capitalize two-words" /><integer name="index" │ │
│ │ format="1-indexed" /></object>` => `{'baz': {'foo': 'Some String', 'index': 1}}` │ │
│ │ │ │
│ │ │ │
│ │ │ │
│ │ Json Output: │ │
│ │ │ │
│ │ │ │
│ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │
│ ╭──────────────────────────────────────────── Message History ────────────────────────────────────────────╮ │
│ │ ┏━━━━━━┳━━━━━━━━━┓ │ │
│ │ ┃ Role ┃ Content ┃ │ │
│ │ ┡━━━━━━╇━━━━━━━━━┩ │ │
│ │ └──────┴─────────┘ │ │
│ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │
│ ╭──────────────────────────────────────────── Raw LLM Output ─────────────────────────────────────────────╮ │
│ │ { │ │
│ │ "pet_type": "dog", │ │
│ │ "name": "FuzzyBear", │ │
│ │ "breed": { │ │
│ │ "breed_name": "Labrador Retriever", │ │
│ │ "common_colors": ["black", "yellow", "chocolate"], │ │
│ │ "life_expectancy": "13 to 16 years" │ │
│ │ } │ │
│ │ } │ │
│ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │
│ ╭─────────────────────────────────────────── Validated Output ────────────────────────────────────────────╮ │
│ │ { │ │
│ │ 'pet_type': 'dog', │ │
│ │ 'name': 'FuzzyBear', │ │
│ │ 'breed': { │ │
│ │ 'breed_name': 'Labrador Retriever', │ │
│ │ 'common_colors': ['black', 'yellow', 'chocolate'], │ │
│ │ 'life_expectancy': '13 to 16 years' │ │
│ │ } │ │
│ │ } │ │
│ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────╯