Getting Started
In this notebook, we will go through the basics of creating a RAIL spec and using Guardrails to enforce it.
Objective
Our goal is to extract detailed patient information from a medical record. We will use a simple example to demonstrate how Guardrails can be used to enforce a RAIL spec.
As an example, we will use the following medical record:
We want our extracted information to contain the following fields:
- Patient's gender
- Patient's age
- A list of symptoms, each with a severity rating and an affected area
- A list of medications, each with information about the patient's response to the medication
Installation
To get started, install the guardrails-ai package with pip. Also, install pydantic, typing, openai, and rich for this example.
Creating a RAIL spec
At the heart of Guardrails is the RAIL spec.
RAIL started as a flavor of XML (standing for Reliable AI markup Language) that describes the expected structure and type of the output of the LLM, the quality criteria for the output to be valid and corrective actions to be taken if the output is invalid. Rails can be written as Pydantic as well. In this example, we'll be developing a Guard using Pydantic.
A RAIL is composed of 2 main components:
- Output schema
- Prompt
Output schema
We start by creating the output schema. The output schema is a Pydantic object that describes the expected structure and type of the output of the LLM.
Specifying structure
As mentioned earlier, we want our extracted information to contain the following fields: gender, age, symptoms, and medications. Using a Pydantic RAIL, we can create fairly complex output schemas. For example, we can specify that a field is a list of objects, etc. We see an example of this below.
RAIL Spec
from pydantic import BaseModel, Field
from typing import List
class Symptom(BaseModel):
symptom: str = Field(description="Symptom that a patient is experiencing")
affected_area: str= Field(description="What part of the body the symptom is affecting")
class Medication(BaseModel):
medication: str = Field(description="Name of the medication the patient is taking")
response: str = Field(description="How the patient is responding to the medication")
class PatientInfo(BaseModel):
gender: str = Field(description="Patient's gender")
age: int = Field(description="Patient's age")
symptoms: List[Symptom] = Field(description="Symptoms that the patient is currently experiencing. Each symptom should be classified into separate item in the list.")
current_meds: List[Medication] = Field(description="Medications the patient is currently taking and their response")
Specifying quality criteria
Next, we want to specify the quality criteria for the output to be valid and corrective actions to be taken if the output is invalid. We can do this by adding a format tag to each field in the output schema. Format tags can either be enforced by Guardrails, or they can only be suggetions to the LLM. You can see the list of validators enforced by Guardrails here. Additionally, you can create your own custom validators, see examples here 1, 2, 3.
As an example, for our use case we specify that the affected_area of symptoms should be one of the following: ['head', 'neck', 'chest']. For this, we use the valid-choices validator.
Specifying corrective actions
Additionally, we can specify how to handle invalid output by adding a on-fail-... attribute to an element.
For example, we can specify that if the affected_area of a symptom is not one of the valid choices, we should re-prompt the LLM to correct its output.
We do this by adding the on-fail-valid-choices='reask' attribute to the affected_area field. To see the full list of corrective actions, see here.
Finally, our updated output schema looks like:
from guardrails.validators import ValidRange, ValidChoices
class Symptom(BaseModel):
symptom: str = Field(description="Symptom that a patient is experiencing")
affected_area: str= Field(
description="What part of the body the symptom is affecting",
#(2)!
validators=[ValidChoices(choices=['head', 'neck', 'chest'], on_fail='reask')]
)
class Medication(BaseModel):
medication: str = Field(description="Name of the medication the patient is taking")
response: str = Field(description="How the patient is responding to the medication")
class PatientInfo(BaseModel):
gender: str = Field(description="Patient's gender")
age: int = Field(
description="Patient's age",
#(1)!
validators=[ValidRange(min=0, max=100)]
)
symptoms: List[Symptom] = Field(description="Symptoms that the patient is currently experiencing. Each symptom should be classified into separate item in the list.")
current_meds: List[Medication] = Field(description="Medications the patient is currently taking and their response")
- We specify the
valid-rangevalidator for theagefield. This validator takes the min and max arguments, which we pass as0and100respectively. - We specify the
valid-choicesvalidator for theaffected_areafield. This validator will check that the value of the field is one of the choices specified in theformatattribute. We specify that if theaffected_areafield fails thevalid-choicesvalidator, we should re-prompt the LLM to correct its output. This is done by setting theon-fail-valid-choicesattribute toreask.
Note
In order to pass in a list of choices, we wrap the list in curly braces -- {['head', 'neck', 'chest']}. Anything inside the curly braces will be evaluated as a Python expression. This allows us to pass in more complex data structures or objects as arguments to the validator.
Prompt
The prompt contains high level instructions for the LLM, as well as any parameters that need to be substituted at runtime.
For our use case, we create the following prompt:
prompt = """
Given the following doctor's notes about a patient,
please extract a dictionary that contains the patient's information. <!-- (1)! -->
${doctors_notes} <!-- (2)! -->
${gr.complete_json_suffix_v2} <!-- (3)! -->
"""
- This is the high level instructions for the LLM.
- This is a prompt parameter that will be substituted at runtime. We will see how to do this in the next section.
- This is some boilerplate text that formats helps the LLM output a valid JSON object. Guardrails will automatically add this text to the end of the LLM's output. You can see the full list of boilerplate text here.
Full rail spec
Putting it all together, you can see what the full rail spec for our use case looks like:
from pydantic import BaseModel, Field
from typing import List, Optional
from guardrails.validators import ValidRange, ValidChoices
prompt = """
Given the following doctor's notes about a patient, please extract a dictionary that contains the patient's information.
${doctors_notes}
${gr.complete_json_suffix_v2}
"""
class Symptom(BaseModel):
symptom: str = Field(description="Symptom that a patient is experiencing")
affected_area: str = Field(description="What part of the body the symptom is affecting", validators=[ValidChoices(choices=['head', 'neck', 'chest'], on_fail="reask")])
class Medication(BaseModel):
medication: str = Field(description="Name of the medication the patient is taking")
response: str = Field(description="How the patient is responding to the medication")
class PatientInfo(BaseModel):
gender: str = Field(description="Patient's gender")
age: int = Field(validators=[ValidRange(min=0, max=100, on_fail="fix")])
symptoms: List[Symptom] = Field(description="Symptoms that the patient is currently experiencing. Each symptom should be classified into a separate item in the list.")
current_meds: List[Medication] = Field(description="Medications the patient is currently taking and their response")
This can also be written using the actual XML based rail spec, as found here.
Using Guardrails to enforce the RAIL spec
We use the RAIL spec to create a Guard object. The Guard object is used to wrap the LLM API call and enforce the RAIL spec on the output of the LLM call.
We can see that the Guard object compiles the RAIL output specification and adds it to the provided prompt.
Next, we call the Guard object with the LLM API call as the first argument and add any additional arguments to the LLM API call as the remaining arguments.
import openai
import os
# Wrap the OpenAI API call with the `guard` object
raw_llm_output, validated_output = guard(
openai.Completion.create,
prompt_params={"doctors_notes": doctors_notes},
engine="text-davinci-003",
max_tokens=1024,
temperature=0.3,
)
# Print the validated output from the LLM
print(validated_output)
Async event loop found, but guard was invoked synchronously.For validator parallelization, please call `validate_async` instead.
Async event loop found, but guard was invoked synchronously.For validator parallelization, please call `validate_async` instead.
{ 'gender': 'Male', 'age': 49, 'symptoms': [ {'symptom': 'macular rash', 'affected_area': 'head'}, {'symptom': 'itchy', 'affected_area': 'neck'}, {'symptom': 'flaky', 'affected_area': 'chest'}, {'symptom': 'slightly scaly', 'affected_area': 'chest'} ], 'current_meds': [{'medication': 'OTC steroid cream', 'response': 'Moderate response'}] }
We can see that our output was structured correctly, and that the values of the affected_area field were one of the valid choices.
Inspecting logs to peek under the hood
Guardrails logs all of the steps it takes to enforce the RAIL spec. We inspect these logs below to see all the steps Guardrails took to enforce the RAIL spec.
Inspecting Step 1: Getting LLM output
╭──────────────────────────────────────────────────── Prompt ─────────────────────────────────────────────────────╮ │ │ │ Given the following doctor's notes about a patient, please extract a dictionary that contains the patient's │ │ information. │ │ │ │ 49 y/o Male with chronic macular rash to face & hair, worse in beard, eyebrows & nares. │ │ Itchy, flaky, slightly scaly. Moderate response to OTC steroid cream │ │ │ │ │ │ Given below is XML that describes the information to extract from this document and the tags to extract it │ │ into. │ │ │ │ <output> │ │ <string name="gender" description="Patient's gender"/> │ │ <integer name="age" format="valid-range: min=0 max=100"/> │ │ <list name="symptoms" description="Symptoms that the patient is currently experiencing. Each symptom should │ │ be classified into a separate item in the list."> │ │ <object> │ │ <string name="symptom" description="Symptom that a patient is experiencing"/> │ │ <string name="affected_area" format="valid-choices: choices=['head', 'neck', 'chest']" │ │ description="What part of the body the symptom is affecting"/> │ │ </object> │ │ </list> │ │ <list name="current_meds" description="Medications the patient is currently taking and their response"> │ │ <object> │ │ <string name="medication" description="Name of the medication the patient is taking"/> │ │ <string name="response" description="How the patient is responding to the medication"/> │ │ </object> │ │ </list> │ │ </output> │ │ │ │ │ │ ONLY return a valid JSON object (no other text is necessary), where the key of the field in JSON is the `name` │ │ attribute of the corresponding XML, and the value is of the type specified by the corresponding XML's tag. The │ │ JSON MUST conform to the XML format, including any types and format requests e.g. requests for lists, objects │ │ and specific types. Be correct and concise. │ │ │ │ Here are examples of simple (XML, JSON) pairs that show the expected behavior: │ │ - `<string name='foo' format='two-words lower-case' />` => `{'foo': 'example one'}` │ │ - `<list name='bar'><string format='upper-case' /></list>` => `{"bar": ['STRING ONE', 'STRING TWO', etc.]}` │ │ - `<object name='baz'><string name="foo" format="capitalize two-words" /><integer name="index" │ │ format="1-indexed" /></object>` => `{'baz': {'foo': 'Some String', 'index': 1}}` │ │ │ │ │ │ │ │ Json Output: │ │ │ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ ╭──────────────────────────────────────────────── Message History ────────────────────────────────────────────────╮ │ ┏━━━━━━┳━━━━━━━━━┓ │ │ ┃ Role ┃ Content ┃ │ │ ┡━━━━━━╇━━━━━━━━━┩ │ │ └──────┴─────────┘ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ ╭──────────────────────────────────────────────── Raw LLM Output ─────────────────────────────────────────────────╮ │ { │ │ "gender": "Male", │ │ "age": 49, │ │ "symptoms": [ │ │ { │ │ "symptom": "macular rash", │ │ "affected_area": "face & hair" │ │ }, │ │ { │ │ "symptom": "itchy", │ │ "affected_area": "beard, eyebrows & nares" │ │ }, │ │ { │ │ "symptom": "flaky", │ │ "affected_area": "beard, eyebrows & nares" │ │ }, │ │ { │ │ "symptom": "slightly scaly", │ │ "affected_area": "beard, eyebrows & nares" │ │ } │ │ ], │ │ "current_meds": [ │ │ { │ │ "medication": "OTC steroid cream", │ │ "response": "Moderate response" │ │ } │ │ ] │ │ } │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ ╭─────────────────────────────────────────────── Validated Output ────────────────────────────────────────────────╮ │ { │ │ 'gender': 'Male', │ │ 'age': 49, │ │ 'symptoms': [ │ │ { │ │ 'symptom': 'macular rash', │ │ 'affected_area': FieldReAsk( │ │ incorrect_value='face & hair', │ │ fail_results=[ │ │ FailResult( │ │ outcome='fail', │ │ metadata=None, │ │ error_message="Value face & hair is not in choices ['head', 'neck', 'chest'].", │ │ fix_value=None │ │ ) │ │ ], │ │ path=['symptoms', 0, 'affected_area'] │ │ ) │ │ }, │ │ { │ │ 'symptom': 'itchy', │ │ 'affected_area': FieldReAsk( │ │ incorrect_value='beard, eyebrows & nares', │ │ fail_results=[ │ │ FailResult( │ │ outcome='fail', │ │ metadata=None, │ │ error_message="Value beard, eyebrows & nares is not in choices ['head', 'neck', │ │ 'chest'].", │ │ fix_value=None │ │ ) │ │ ], │ │ path=['symptoms', 1, 'affected_area'] │ │ ) │ │ }, │ │ { │ │ 'symptom': 'flaky', │ │ 'affected_area': FieldReAsk( │ │ incorrect_value='beard, eyebrows & nares', │ │ fail_results=[ │ │ FailResult( │ │ outcome='fail', │ │ metadata=None, │ │ error_message="Value beard, eyebrows & nares is not in choices ['head', 'neck', │ │ 'chest'].", │ │ fix_value=None │ │ ) │ │ ], │ │ path=['symptoms', 2, 'affected_area'] │ │ ) │ │ }, │ │ { │ │ 'symptom': 'slightly scaly', │ │ 'affected_area': FieldReAsk( │ │ incorrect_value='beard, eyebrows & nares', │ │ fail_results=[ │ │ FailResult( │ │ outcome='fail', │ │ metadata=None, │ │ error_message="Value beard, eyebrows & nares is not in choices ['head', 'neck', │ │ 'chest'].", │ │ fix_value=None │ │ ) │ │ ], │ │ path=['symptoms', 3, 'affected_area'] │ │ ) │ │ } │ │ ], │ │ 'current_meds': [ │ │ {'medication': 'OTC steroid cream', 'response': 'Moderate response'} │ │ ] │ │ } │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
We can see that the output returned by the LLM contained invalid values for the affected_area field.
On validation, Guardrails created ReAsk objects that contains all of the information needed to re-prompt the LLM to correct its output.
Inspecting Step 2: Re-prompting LLM
╭──────────────────────────────────────────────────── Prompt ─────────────────────────────────────────────────────╮ │ │ │ I was given the following JSON response, which had problems due to incorrect values. │ │ │ │ { │ │ "gender": "Male", │ │ "age": 49, │ │ "symptoms": [ │ │ { │ │ "symptom": "macular rash", │ │ "affected_area": { │ │ "incorrect_value": "face & hair", │ │ "error_messages": [ │ │ "Value face & hair is not in choices ['head', 'neck', 'chest']." │ │ ] │ │ } │ │ }, │ │ { │ │ "symptom": "itchy", │ │ "affected_area": { │ │ "incorrect_value": "beard, eyebrows & nares", │ │ "error_messages": [ │ │ "Value beard, eyebrows & nares is not in choices ['head', 'neck', 'chest']." │ │ ] │ │ } │ │ }, │ │ { │ │ "symptom": "flaky", │ │ "affected_area": { │ │ "incorrect_value": "beard, eyebrows & nares", │ │ "error_messages": [ │ │ "Value beard, eyebrows & nares is not in choices ['head', 'neck', 'chest']." │ │ ] │ │ } │ │ }, │ │ { │ │ "symptom": "slightly scaly", │ │ "affected_area": { │ │ "incorrect_value": "beard, eyebrows & nares", │ │ "error_messages": [ │ │ "Value beard, eyebrows & nares is not in choices ['head', 'neck', 'chest']." │ │ ] │ │ } │ │ } │ │ ], │ │ "current_meds": [ │ │ { │ │ "medication": "OTC steroid cream", │ │ "response": "Moderate response" │ │ } │ │ ] │ │ } │ │ │ │ Help me correct the incorrect values based on the given error messages. │ │ │ │ Given below is XML that describes the information to extract from this document and the tags to extract it │ │ into. │ │ │ │ <output> │ │ <string name="gender" description="Patient's gender"/> │ │ <integer name="age" format="valid-range: min=0 max=100"/> │ │ <list name="symptoms" description="Symptoms that the patient is currently experiencing. Each symptom should │ │ be classified into a separate item in the list."> │ │ <object> │ │ <string name="symptom" description="Symptom that a patient is experiencing"/> │ │ <string name="affected_area" format="valid-choices: choices=['head', 'neck', 'chest']" │ │ description="What part of the body the symptom is affecting"/> │ │ </object> │ │ </list> │ │ <list name="current_meds" description="Medications the patient is currently taking and their response"> │ │ <object> │ │ <string name="medication" description="Name of the medication the patient is taking"/> │ │ <string name="response" description="How the patient is responding to the medication"/> │ │ </object> │ │ </list> │ │ </output> │ │ │ │ │ │ ONLY return a valid JSON object (no other text is necessary), where the key of the field in JSON is the `name` │ │ attribute of the corresponding XML, and the value is of the type specified by the corresponding XML's tag. The │ │ JSON MUST conform to the XML format, including any types and format requests e.g. requests for lists, objects │ │ and specific types. Be correct and concise. If you are unsure anywhere, enter `null`. │ │ │ │ │ │ Json Output: │ │ │ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ ╭───────────────────────────────────────────────── Instructions ──────────────────────────────────────────────────╮ │ │ │ You are a helpful assistant only capable of communicating with valid JSON, and no other text. │ │ │ │ ONLY return a valid JSON object (no other text is necessary), where the key of the field in JSON is the `name` │ │ attribute of the corresponding XML, and the value is of the type specified by the corresponding XML's tag. The │ │ JSON MUST conform to the XML format, including any types and format requests e.g. requests for lists, objects │ │ and specific types. Be correct and concise. If you are unsure anywhere, enter `null`. │ │ │ │ Here are examples of simple (XML, JSON) pairs that show the expected behavior: │ │ - `<string name='foo' format='two-words lower-case' />` => `{'foo': 'example one'}` │ │ - `<list name='bar'><string format='upper-case' /></list>` => `{"bar": ['STRING ONE', 'STRING TWO', etc.]}` │ │ - `<object name='baz'><string name="foo" format="capitalize two-words" /><integer name="index" │ │ format="1-indexed" /></object>` => `{'baz': {'foo': 'Some String', 'index': 1}}` │ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ ╭──────────────────────────────────────────────── Message History ────────────────────────────────────────────────╮ │ No message history. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ ╭──────────────────────────────────────────────── Raw LLM Output ─────────────────────────────────────────────────╮ │ { │ │ "gender": "Male", │ │ "age": 49, │ │ "symptoms": [ │ │ { │ │ "symptom": "macular rash", │ │ "affected_area": "head" │ │ }, │ │ { │ │ "symptom": "itchy", │ │ "affected_area": "neck" │ │ }, │ │ { │ │ "symptom": "flaky", │ │ "affected_area": "chest" │ │ }, │ │ { │ │ "symptom": "slightly scaly", │ │ "affected_area": "chest" │ │ } │ │ ], │ │ "current_meds": [ │ │ { │ │ "medication": "OTC steroid cream", │ │ "response": "Moderate response" │ │ } │ │ ] │ │ } │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ ╭─────────────────────────────────────────────── Validated Output ────────────────────────────────────────────────╮ │ { │ │ 'gender': 'Male', │ │ 'age': 49, │ │ 'symptoms': [ │ │ {'symptom': 'macular rash', 'affected_area': 'head'}, │ │ {'symptom': 'itchy', 'affected_area': 'neck'}, │ │ {'symptom': 'flaky', 'affected_area': 'chest'}, │ │ {'symptom': 'slightly scaly', 'affected_area': 'chest'} │ │ ], │ │ 'current_meds': [ │ │ {'medication': 'OTC steroid cream', 'response': 'Moderate response'} │ │ ] │ │ } │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
The prompt to the LLM was fully constructed by Guardrails. Additionally, the prompt only contained the previously invalid values for the affected_area field.
On receiving the corrected LLM output, Guardrails merges it with the previously invalid values to create a valid output. It then reruns the validation step to ensure that the output is valid, and returns the output to the user.
Bonus: See the full history of calls
Guardrails also logs the full history of calls made to the LLM. This can be useful for debugging purposes.
Logs ├── ╭────────────────────────────────────────────────── Step 0 ───────────────────────────────────────────────────╮ │ │ ╭──────────────────────────────────────────────── Prompt ─────────────────────────────────────────────────╮ │ │ │ │ │ │ │ │ │ Given the following doctor's notes about a patient, please extract a dictionary that contains the │ │ │ │ │ patient's information. │ │ │ │ │ │ │ │ │ │ 49 y/o Male with chronic macular rash to face & hair, worse in beard, eyebrows & nares. │ │ │ │ │ Itchy, flaky, slightly scaly. Moderate response to OTC steroid cream │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ Given below is XML that describes the information to extract from this document and the tags to extract │ │ │ │ │ it into. │ │ │ │ │ │ │ │ │ │ <output> │ │ │ │ │ <string name="gender" description="Patient's gender"/> │ │ │ │ │ <integer name="age" format="valid-range: min=0 max=100"/> │ │ │ │ │ <list name="symptoms" description="Symptoms that the patient is currently experiencing. Each │ │ │ │ │ symptom should be classified into a separate item in the list."> │ │ │ │ │ <object> │ │ │ │ │ <string name="symptom" description="Symptom that a patient is experiencing"/> │ │ │ │ │ <string name="affected_area" format="valid-choices: choices=['head', 'neck', 'chest']" │ │ │ │ │ description="What part of the body the symptom is affecting"/> │ │ │ │ │ </object> │ │ │ │ │ </list> │ │ │ │ │ <list name="current_meds" description="Medications the patient is currently taking and their │ │ │ │ │ response"> │ │ │ │ │ <object> │ │ │ │ │ <string name="medication" description="Name of the medication the patient is taking"/> │ │ │ │ │ <string name="response" description="How the patient is responding to the medication"/> │ │ │ │ │ </object> │ │ │ │ │ </list> │ │ │ │ │ </output> │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ ONLY return a valid JSON object (no other text is necessary), where the key of the field in JSON is the │ │ │ │ │ `name` attribute of the corresponding XML, and the value is of the type specified by the corresponding │ │ │ │ │ XML's tag. The JSON MUST conform to the XML format, including any types and format requests e.g. │ │ │ │ │ requests for lists, objects and specific types. Be correct and concise. │ │ │ │ │ │ │ │ │ │ Here are examples of simple (XML, JSON) pairs that show the expected behavior: │ │ │ │ │ - `<string name='foo' format='two-words lower-case' />` => `{'foo': 'example one'}` │ │ │ │ │ - `<list name='bar'><string format='upper-case' /></list>` => `{"bar": ['STRING ONE', 'STRING TWO', │ │ │ │ │ etc.]}` │ │ │ │ │ - `<object name='baz'><string name="foo" format="capitalize two-words" /><integer name="index" │ │ │ │ │ format="1-indexed" /></object>` => `{'baz': {'foo': 'Some String', 'index': 1}}` │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ Json Output: │ │ │ │ │ │ │ │ │ │ │ │ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │ │ │ ╭──────────────────────────────────────────── Message History ────────────────────────────────────────────╮ │ │ │ │ ┏━━━━━━┳━━━━━━━━━┓ │ │ │ │ │ ┃ Role ┃ Content ┃ │ │ │ │ │ ┡━━━━━━╇━━━━━━━━━┩ │ │ │ │ │ └──────┴─────────┘ │ │ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │ │ │ ╭──────────────────────────────────────────── Raw LLM Output ─────────────────────────────────────────────╮ │ │ │ │ { │ │ │ │ │ "gender": "Male", │ │ │ │ │ "age": 49, │ │ │ │ │ "symptoms": [ │ │ │ │ │ { │ │ │ │ │ "symptom": "macular rash", │ │ │ │ │ "affected_area": "face & hair" │ │ │ │ │ }, │ │ │ │ │ { │ │ │ │ │ "symptom": "itchy", │ │ │ │ │ "affected_area": "beard, eyebrows & nares" │ │ │ │ │ }, │ │ │ │ │ { │ │ │ │ │ "symptom": "flaky", │ │ │ │ │ "affected_area": "beard, eyebrows & nares" │ │ │ │ │ }, │ │ │ │ │ { │ │ │ │ │ "symptom": "slightly scaly", │ │ │ │ │ "affected_area": "beard, eyebrows & nares" │ │ │ │ │ } │ │ │ │ │ ], │ │ │ │ │ "current_meds": [ │ │ │ │ │ { │ │ │ │ │ "medication": "OTC steroid cream", │ │ │ │ │ "response": "Moderate response" │ │ │ │ │ } │ │ │ │ │ ] │ │ │ │ │ } │ │ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │ │ │ ╭─────────────────────────────────────────── Validated Output ────────────────────────────────────────────╮ │ │ │ │ { │ │ │ │ │ 'gender': 'Male', │ │ │ │ │ 'age': 49, │ │ │ │ │ 'symptoms': [ │ │ │ │ │ { │ │ │ │ │ 'symptom': 'macular rash', │ │ │ │ │ 'affected_area': FieldReAsk( │ │ │ │ │ incorrect_value='face & hair', │ │ │ │ │ fail_results=[ │ │ │ │ │ FailResult( │ │ │ │ │ outcome='fail', │ │ │ │ │ metadata=None, │ │ │ │ │ error_message="Value face & hair is not in choices ['head', 'neck', 'chest'].", │ │ │ │ │ fix_value=None │ │ │ │ │ ) │ │ │ │ │ ], │ │ │ │ │ path=['symptoms', 0, 'affected_area'] │ │ │ │ │ ) │ │ │ │ │ }, │ │ │ │ │ { │ │ │ │ │ 'symptom': 'itchy', │ │ │ │ │ 'affected_area': FieldReAsk( │ │ │ │ │ incorrect_value='beard, eyebrows & nares', │ │ │ │ │ fail_results=[ │ │ │ │ │ FailResult( │ │ │ │ │ outcome='fail', │ │ │ │ │ metadata=None, │ │ │ │ │ error_message="Value beard, eyebrows & nares is not in choices ['head', 'neck', │ │ │ │ │ 'chest'].", │ │ │ │ │ fix_value=None │ │ │ │ │ ) │ │ │ │ │ ], │ │ │ │ │ path=['symptoms', 1, 'affected_area'] │ │ │ │ │ ) │ │ │ │ │ }, │ │ │ │ │ { │ │ │ │ │ 'symptom': 'flaky', │ │ │ │ │ 'affected_area': FieldReAsk( │ │ │ │ │ incorrect_value='beard, eyebrows & nares', │ │ │ │ │ fail_results=[ │ │ │ │ │ FailResult( │ │ │ │ │ outcome='fail', │ │ │ │ │ metadata=None, │ │ │ │ │ error_message="Value beard, eyebrows & nares is not in choices ['head', 'neck', │ │ │ │ │ 'chest'].", │ │ │ │ │ fix_value=None │ │ │ │ │ ) │ │ │ │ │ ], │ │ │ │ │ path=['symptoms', 2, 'affected_area'] │ │ │ │ │ ) │ │ │ │ │ }, │ │ │ │ │ { │ │ │ │ │ 'symptom': 'slightly scaly', │ │ │ │ │ 'affected_area': FieldReAsk( │ │ │ │ │ incorrect_value='beard, eyebrows & nares', │ │ │ │ │ fail_results=[ │ │ │ │ │ FailResult( │ │ │ │ │ outcome='fail', │ │ │ │ │ metadata=None, │ │ │ │ │ error_message="Value beard, eyebrows & nares is not in choices ['head', 'neck', │ │ │ │ │ 'chest'].", │ │ │ │ │ fix_value=None │ │ │ │ │ ) │ │ │ │ │ ], │ │ │ │ │ path=['symptoms', 3, 'affected_area'] │ │ │ │ │ ) │ │ │ │ │ } │ │ │ │ │ ], │ │ │ │ │ 'current_meds': [ │ │ │ │ │ {'medication': 'OTC steroid cream', 'response': 'Moderate response'} │ │ │ │ │ ] │ │ │ │ │ } │ │ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ └── ╭────────────────────────────────────────────────── Step 1 ───────────────────────────────────────────────────╮ │ ╭──────────────────────────────────────────────── Prompt ─────────────────────────────────────────────────╮ │ │ │ │ │ │ │ I was given the following JSON response, which had problems due to incorrect values. │ │ │ │ │ │ │ │ { │ │ │ │ "gender": "Male", │ │ │ │ "age": 49, │ │ │ │ "symptoms": [ │ │ │ │ { │ │ │ │ "symptom": "macular rash", │ │ │ │ "affected_area": { │ │ │ │ "incorrect_value": "face & hair", │ │ │ │ "error_messages": [ │ │ │ │ "Value face & hair is not in choices ['head', 'neck', 'chest']." │ │ │ │ ] │ │ │ │ } │ │ │ │ }, │ │ │ │ { │ │ │ │ "symptom": "itchy", │ │ │ │ "affected_area": { │ │ │ │ "incorrect_value": "beard, eyebrows & nares", │ │ │ │ "error_messages": [ │ │ │ │ "Value beard, eyebrows & nares is not in choices ['head', 'neck', 'chest']." │ │ │ │ ] │ │ │ │ } │ │ │ │ }, │ │ │ │ { │ │ │ │ "symptom": "flaky", │ │ │ │ "affected_area": { │ │ │ │ "incorrect_value": "beard, eyebrows & nares", │ │ │ │ "error_messages": [ │ │ │ │ "Value beard, eyebrows & nares is not in choices ['head', 'neck', 'chest']." │ │ │ │ ] │ │ │ │ } │ │ │ │ }, │ │ │ │ { │ │ │ │ "symptom": "slightly scaly", │ │ │ │ "affected_area": { │ │ │ │ "incorrect_value": "beard, eyebrows & nares", │ │ │ │ "error_messages": [ │ │ │ │ "Value beard, eyebrows & nares is not in choices ['head', 'neck', 'chest']." │ │ │ │ ] │ │ │ │ } │ │ │ │ } │ │ │ │ ], │ │ │ │ "current_meds": [ │ │ │ │ { │ │ │ │ "medication": "OTC steroid cream", │ │ │ │ "response": "Moderate response" │ │ │ │ } │ │ │ │ ] │ │ │ │ } │ │ │ │ │ │ │ │ Help me correct the incorrect values based on the given error messages. │ │ │ │ │ │ │ │ Given below is XML that describes the information to extract from this document and the tags to extract │ │ │ │ it into. │ │ │ │ │ │ │ │ <output> │ │ │ │ <string name="gender" description="Patient's gender"/> │ │ │ │ <integer name="age" format="valid-range: min=0 max=100"/> │ │ │ │ <list name="symptoms" description="Symptoms that the patient is currently experiencing. Each │ │ │ │ symptom should be classified into a separate item in the list."> │ │ │ │ <object> │ │ │ │ <string name="symptom" description="Symptom that a patient is experiencing"/> │ │ │ │ <string name="affected_area" format="valid-choices: choices=['head', 'neck', 'chest']" │ │ │ │ description="What part of the body the symptom is affecting"/> │ │ │ │ </object> │ │ │ │ </list> │ │ │ │ <list name="current_meds" description="Medications the patient is currently taking and their │ │ │ │ response"> │ │ │ │ <object> │ │ │ │ <string name="medication" description="Name of the medication the patient is taking"/> │ │ │ │ <string name="response" description="How the patient is responding to the medication"/> │ │ │ │ </object> │ │ │ │ </list> │ │ │ │ </output> │ │ │ │ │ │ │ │ │ │ │ │ ONLY return a valid JSON object (no other text is necessary), where the key of the field in JSON is the │ │ │ │ `name` attribute of the corresponding XML, and the value is of the type specified by the corresponding │ │ │ │ XML's tag. The JSON MUST conform to the XML format, including any types and format requests e.g. │ │ │ │ requests for lists, objects and specific types. Be correct and concise. If you are unsure anywhere, │ │ │ │ enter `null`. │ │ │ │ │ │ │ │ │ │ │ │ Json Output: │ │ │ │ │ │ │ │ │ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │ │ ╭───────────────────────────────────────────── Instructions ──────────────────────────────────────────────╮ │ │ │ │ │ │ │ You are a helpful assistant only capable of communicating with valid JSON, and no other text. │ │ │ │ │ │ │ │ ONLY return a valid JSON object (no other text is necessary), where the key of the field in JSON is the │ │ │ │ `name` attribute of the corresponding XML, and the value is of the type specified by the corresponding │ │ │ │ XML's tag. The JSON MUST conform to the XML format, including any types and format requests e.g. │ │ │ │ requests for lists, objects and specific types. Be correct and concise. If you are unsure anywhere, │ │ │ │ enter `null`. │ │ │ │ │ │ │ │ Here are examples of simple (XML, JSON) pairs that show the expected behavior: │ │ │ │ - `<string name='foo' format='two-words lower-case' />` => `{'foo': 'example one'}` │ │ │ │ - `<list name='bar'><string format='upper-case' /></list>` => `{"bar": ['STRING ONE', 'STRING TWO', │ │ │ │ etc.]}` │ │ │ │ - `<object name='baz'><string name="foo" format="capitalize two-words" /><integer name="index" │ │ │ │ format="1-indexed" /></object>` => `{'baz': {'foo': 'Some String', 'index': 1}}` │ │ │ │ │ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │ │ ╭──────────────────────────────────────────── Message History ────────────────────────────────────────────╮ │ │ │ No message history. │ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │ │ ╭──────────────────────────────────────────── Raw LLM Output ─────────────────────────────────────────────╮ │ │ │ { │ │ │ │ "gender": "Male", │ │ │ │ "age": 49, │ │ │ │ "symptoms": [ │ │ │ │ { │ │ │ │ "symptom": "macular rash", │ │ │ │ "affected_area": "head" │ │ │ │ }, │ │ │ │ { │ │ │ │ "symptom": "itchy", │ │ │ │ "affected_area": "neck" │ │ │ │ }, │ │ │ │ { │ │ │ │ "symptom": "flaky", │ │ │ │ "affected_area": "chest" │ │ │ │ }, │ │ │ │ { │ │ │ │ "symptom": "slightly scaly", │ │ │ │ "affected_area": "chest" │ │ │ │ } │ │ │ │ ], │ │ │ │ "current_meds": [ │ │ │ │ { │ │ │ │ "medication": "OTC steroid cream", │ │ │ │ "response": "Moderate response" │ │ │ │ } │ │ │ │ ] │ │ │ │ } │ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │ │ ╭─────────────────────────────────────────── Validated Output ────────────────────────────────────────────╮ │ │ │ { │ │ │ │ 'gender': 'Male', │ │ │ │ 'age': 49, │ │ │ │ 'symptoms': [ │ │ │ │ {'symptom': 'macular rash', 'affected_area': 'head'}, │ │ │ │ {'symptom': 'itchy', 'affected_area': 'neck'}, │ │ │ │ {'symptom': 'flaky', 'affected_area': 'chest'}, │ │ │ │ {'symptom': 'slightly scaly', 'affected_area': 'chest'} │ │ │ │ ], │ │ │ │ 'current_meds': [ │ │ │ │ {'medication': 'OTC steroid cream', 'response': 'Moderate response'} │ │ │ │ ] │ │ │ │ } │ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────╯