Using Guardrails with Chat Models
Note
To download this example as a Jupyter notebook, click here.
In this example, we will set up Guardrails with a chat model.
Objective
We retry the entity extraction example using a chat model.
Step 0: Download PDF and load it as string
To get started, download the document from here and save it in data/chase_card_agreement.pdf.
Guardrails has some built-in functions to help with common tasks. Here, we will use the read_pdf function to load the PDF as a string.
Chase Credit Card Document: 2/25/23, 7:59 PM about:blank about:blank 1/4 PRICING INFORMATION INTEREST RATES AND INTEREST CHARGES Purchase Annual Percentage Rate (APR) 0% Intro APR for the first 18 months that your Account is open. After that, 19.49%. This APR will vary with the market based on the Prim ...
Step 1: Create the RAIL Spec with <instructions> tags
In order to use Guardrails with a chat model, we need to add <instructions> tags to the RAIL spec. These tags will be used to generate the system message for the chat model.
Ordinarily, everything that is contained in the <prompt> tag will be split across <prompt> and <instructions> tags. Here's an example illustrating the differences.
```xml
${gr.json_suffix_prompt_examples}
null.
${document}
Extract information from this document and return a JSON that follows the correct schema.
${gr.xml_prefix_prompt}
${output_schema}
</prompt>
```
instructions = """You are a helpful assistant only capable of communicating with valid JSON, and no other text.
${gr.json_suffix_prompt_examples}"""
prompt = """Given the following document, answer the following questions. If the answer doesn't exist in the document, enter `null`.
${document}
Extract information from this document and return a JSON that follows the correct schema.
${gr.xml_prefix_prompt}
${output_schema}"""
After materialization, the two variations of the RAIL specification will look like this when passed to the LLM (regarless if they were initialized from xml or a Pydantic model):
```xml
ONLY return a valid JSON object (no other text is necessary), where the key of the field in JSON is the name attribute of the corresponding XML, and the value is of the type specified by the corresponding XML's tag. The JSON MUST conform to the XML format, including any types and format requests e.g. requests for lists, objects and specific types. Be correct and concise. If you are unsure anywhere, enter null.
Here are examples of simple (XML, JSON) pairs that show the expected behavior:
- <string format="two-words lower-case" name="foo"></string> => {'foo': 'example one'}
- <list name="bar"><string format="upper-case"></string></list> => {"bar": ['STRING ONE', 'STRING TWO', etc.]}
- <object name="baz"><string format="capitalize two-words" name="foo"></string><integer format="1-indexed" name="index"></integer></object> => {'baz': {'foo': 'Some String', 'index': 1}}
null.
${document}
Extract information from this document and return a JSON that follows the correct schema.
Given below is XML that describes the information to extract from this document and the tags to extract it into.
${output_schema}
</prompt>
```
<prompt>
Given the following document, answer the following questions. If the answer doesn't exist in the document, enter `null`.
${document}
Given below is XML that describes the information to extract from this document and the tags to extract it into.
${output_schema}
ONLY return a valid JSON object (no other text is necessary), where the key of the field in JSON is the `name` attribute of the corresponding XML, and the value is of the type specified by the corresponding XML's tag. The JSON MUST conform to the XML format, including any types and format requests e.g. requests for lists, objects and specific types. Be correct and concise. If you are unsure anywhere, enter `null`.
Here are examples of simple (XML, JSON) pairs that show the expected behavior:
- `<string format="two-words lower-case" name="foo"></string>` => `{'foo': 'example one'}`
- `<list name="bar"><string format="upper-case"></string></list>` => `{"bar": ['STRING ONE', 'STRING TWO', etc.]}`
- `<object name="baz"><string format="capitalize two-words" name="foo"></string><integer format="1-indexed" name="index"></integer></object>` => `{'baz': {'foo': 'Some String', 'index': 1}}`
</prompt>
Here's the final RAIL spec as XML:
rail_str = """
<rail version="0.1">
<output>
<list description="What fees and charges are associated with my account?" name="fees">
<object>
<integer format="1-indexed" name="index"></integer>
<string format="lower-case; two-words" name="name" on-fail-lower-case="fix" on-fail-two-words="reask"></string>
<string format="one-line" name="explanation"></string>
<float format="percentage" name="value"></float>
</object>
</list>
<object description="What are the interest rates offered by the bank on savings and checking accounts, loans, and credit products?" name="interest_rates" required="false"></object>
</output>
<instructions>
You are a helpful assistant only capable of communicating with valid JSON, and no other text.
${gr.json_suffix_prompt_examples}
</instructions>
<prompt>
Given the following document, answer the following questions. If the answer doesn't exist in the document, enter
`null`.
${document}
Extract information from this document and return a JSON that follows the correct schema.
${gr.xml_prefix_prompt}
${output_schema}
</prompt>
</rail>
"""
Or using a Pydantic model for the output schema:
from guardrails.validators import LowerCase, TwoWords, OneLine
from pydantic import BaseModel, Field
from typing import List, Optional
instructions = """You are a helpful assistant only capable of communicating with valid JSON, and no other text.
${gr.json_suffix_prompt_examples}"""
prompt = """Given the following document, answer the following questions. If the answer doesn't exist in the document, enter
`null`.
${document}
Extract information from this document and return a JSON that follows the correct schema.
${gr.xml_prefix_prompt}
${output_schema}"""
class Fee(BaseModel):
index: int = Field(validators=[("1-indexed", "noop")])
name: str = Field(validators=[LowerCase(on_fail="fix"), TwoWords(on_fail="reask")])
explanation: str = Field(validators=[OneLine()])
value: float = Field(validators=[("percentage", "noop")])
class CreditCardAgreement(BaseModel):
fees: List[Fee] = Field(description="What fees and charges are associated with my account?")
interest_rates: Optional[dict] = Field(description="What are the interest rates offered by the bank on savings and checking accounts, loans, and credit products?")
Step 2: Create a Guard object with the RAIL Spec
We create a gd.Guard object that will check, validate and correct the output of the LLM. This object:
- Enforces the quality criteria specified in the RAIL spec.
- Takes corrective action when the quality criteria are not met.
- Compiles the schema and type info from the RAIL spec and adds it to the prompt.
Creating the guard from XML:
/Users/calebcourier/Projects/guardrails-ai/guardrails/guardrails/schema.py:218: UserWarning: Validator 1-indexed is not valid for element integer.
warnings.warn(
/Users/calebcourier/Projects/guardrails-ai/guardrails/guardrails/schema.py:218: UserWarning: Validator percentage is not valid for element float.
warnings.warn(
Or from the Pydantic model:
As we can see, a few formatters weren't supported. These formatters won't be enforced in the output, but this information can still be used to generate a prompt.
We see the prompt that will be sent to the LLM. The {document} is substituted with the user provided value at runtime.
Given the following document, answer the following questions. If the answer doesn't exist in the document, enter `null`. ${document} Extract information from this document and return a JSON that follows the correct schema. Given below is XML that describes the information to extract from this document and the tags to extract it into. <output> <list name="fees" description="What fees and charges are associated with my account?"> <object> <integer name="index" format="1-indexed"/> <string name="name" format="lower-case; two-words"/> <string name="explanation" format="one-line"/> <float name="value" format="percentage"/> </object> </list> <object name="interest_rates" description="What are the interest rates offered by the bank on savings and checking accounts, loans, and credit products?"/> </output>
Here's the formatted instructions sent as the system message to the LLM.
You are a helpful assistant only capable of communicating with valid JSON, and no other text. ONLY return a valid JSON object (no other text is necessary), where the key of the field in JSON is the `name` attribute of the corresponding XML, and the value is of the type specified by the corresponding XML's tag. The JSON MUST conform to the XML format, including any types and format requests e.g. requests for lists, objects and specific types. Be correct and concise. If you are unsure anywhere, enter `null`. Here are examples of simple (XML, JSON) pairs that show the expected behavior: - `<string name='foo' format='two-words lower-case' />` => `{'foo': 'example one'}` - `<list name='bar'><string format='upper-case' /></list>` => `{"bar": ['STRING ONE', 'STRING TWO', etc.]}` - `<object name='baz'><string name="foo" format="capitalize two-words" /><integer name="index" format="1-indexed" /></object>` => `{'baz': {'foo': 'Some String', 'index': 1}}`
Step 3: Wrap the LLM API call with Guard
Compared to the entity extraction example, we call the OpenAI ChatCompletion API instead of the OpenAI Completion API.
We also pass the model argument instead of the engine argument.
The guard wrapper returns the raw_llm_respose (which is a simple string), and the validated and corrected output (which is a dictionary).
We can see that the output is a dictionary with the correct schema and types.
SkeletonReAsk( incorrect_value={ 'fees': [ {'index': 1, 'name': 'annual membership fee', 'explanation': 'None', 'value': 0.0}, { 'index': 2, 'name': 'my chase plan fee', 'explanation': 'Monthly fee of 0% of the amount of each eligible purchase transaction or amount selected to create a My Chase Plan while in the 0% Intro Purchase APR period. After that, monthly fee of 1.72% of the amount of each eligible purchase transaction or amount selected to create a My Chase Plan. The My Chase Plan Fee will be determined at the time each My Chase Plan is created and will remain the same until the My Chase Plan is paid in full.', 'value': 1.72 }, { 'index': 3, 'name': 'balance transfers intro fee', 'explanation': 'Either $5 or 3% of the amount of each transfer, whichever is greater, on transfers made within 60 days of account opening. After that: Either $5 or 5% of the amount of each transfer, whichever is greater.', 'value': 5.0 }, { 'index': 4, 'name': 'cash advances', 'explanation': 'Either $10 or 5% of the amount of each transaction, whichever is greater.', 'value': 5.0 }, { 'index': 5, 'name': 'foreign transactions', 'explanation': '3% of the amount of each transaction in U.S. dollars.', 'value': 3.0 }, {'index': 6, 'name': 'late payment', 'explanation': 'Up to $40.', 'value': 40.0}, {'index': 7, 'name': 'over-the-credit-limit', 'explanation': 'None', 'value': None}, {'index': 8, 'name': 'return payment', 'explanation': 'Up to $40.', 'value': 40.0}, {'index': 9, 'name': 'return check', 'explanation': 'None', 'value': None} ], 'interest_rates': None }, fail_results=[ FailResult(outcome='fail', metadata=None, error_message='JSON does not match schema', fix_value=None) ] )
Logs ├── ╭────────────────────────────────────────────────── Step 0 ───────────────────────────────────────────────────╮ │ │ ╭──────────────────────────────────────────────── Prompt ─────────────────────────────────────────────────╮ │ │ │ │ Given the following document, answer the following questions. If the answer doesn't exist in the │ │ │ │ │ document, enter │ │ │ │ │ `null`. │ │ │ │ │ │ │ │ │ │ 2/25/23, 7:59 PM about:blank │ │ │ │ │ about:blank 1/4 │ │ │ │ │ PRICING INFORMATION │ │ │ │ │ INTEREST RATES AND INTEREST CHARGES │ │ │ │ │ Purchase Annual │ │ │ │ │ Percentage Rate (APR) 0% Intro APR for the first 18 months that your Account is open. │ │ │ │ │ After that, 19.49%. This APR will vary with the market based on the Prime │ │ │ │ │ Rate. │ │ │ │ │ a │ │ │ │ │ My Chase Loan │ │ │ │ │ SM APR 19.49%. This APR will vary with the market based on the Prime Rate. │ │ │ │ │ a │ │ │ │ │ Promotional offers with fixed APRs and varying durations may be available from │ │ │ │ │ time to time on some accounts. │ │ │ │ │ Balance Transfer APR 0% Intro APR for the first 18 months that your Account is open. │ │ │ │ │ After that, 19.49%. This APR will vary with the market based on the Prime │ │ │ │ │ Rate. │ │ │ │ │ a │ │ │ │ │ Cash Advance APR 29.49%. This APR will vary with the market based on the Prime Rate. │ │ │ │ │ b │ │ │ │ │ Penalty APR and When │ │ │ │ │ It Applies │ │ │ │ │ Up to 29.99%. This APR will vary with the market based on the Prime Rate. │ │ │ │ │ c │ │ │ │ │ We may apply the Penalty APR to your account if you: │ │ │ │ │ fail to make a Minimum Payment by the date and time that it is due; or │ │ │ │ │ make a payment to us that is returned unpaid. │ │ │ │ │ How Long Will the Penalty APR Apply?: If we apply the Penalty APR for │ │ │ │ │ either of these reasons, the Penalty APR could potentially remain in effect │ │ │ │ │ indefinitely. │ │ │ │ │ How to Avoid Paying │ │ │ │ │ Interest on Purchases │ │ │ │ │ Your due date will be a minimum of 21 days after the close of each billing cycle. │ │ │ │ │ We will not charge you interest on new purchases if you pay your entire balance │ │ │ │ │ or Interest Saving Balance by the due date each month. We will begin charging │ │ │ │ │ interest on balance transfers and cash advances on the transaction date. │ │ │ │ │ Minimum Interest │ │ │ │ │ Charge │ │ │ │ │ None │ │ │ │ │ Credit Card Tips from │ │ │ │ │ the Consumer Financial │ │ │ │ │ Protection Bureau │ │ │ │ │ To learn more about factors to consider when applying for or using a credit card, │ │ │ │ │ visit the website of the Consumer Financial Protection Bureau at │ │ │ │ │ http://www.consumerfinance.gov/learnmore. │ │ │ │ │ FEES │ │ │ │ │ Annual Membership │ │ │ │ │ Fee │ │ │ │ │ None │ │ │ │ │ My Chase Plan │ │ │ │ │ SM Fee │ │ │ │ │ (fixed finance charge) │ │ │ │ │ Monthly fee of 0% of the amount of each eligible purchase transaction or │ │ │ │ │ amount selected to create a My Chase Plan while in the 0% Intro Purchase │ │ │ │ │ APR period. │ │ │ │ │ After that, monthly fee of 1.72% of the amount of each eligible purchase │ │ │ │ │ transaction or amount selected to create a My Chase Plan. The My Chase Plan │ │ │ │ │ Fee will be determined at the time each My Chase Plan is created and will │ │ │ │ │ remain the same until the My Chase Plan is paid in full. │ │ │ │ │ d │ │ │ │ │ Transaction Fees │ │ │ │ │ Balance Transfers Intro fee of either $5 or 3% of the amount of each transfer, whichever is greater, │ │ │ │ │ on transfers made within 60 days of account opening. After that: Either $5 or 5% │ │ │ │ │ of the amount of each transfer, whichever is greater. │ │ │ │ │ Cash Advances Either $10 or 5% of the amount of each transaction, whichever is greater. │ │ │ │ │ 2/25/23, 7:59 PM about:blank │ │ │ │ │ about:blank 2/4 │ │ │ │ │ Foreign Transactions 3% of the amount of each transaction in U.S. dollars. │ │ │ │ │ Penalty Fees │ │ │ │ │ Late Payment Up to $40. │ │ │ │ │ Over-the-Credit-Limit None │ │ │ │ │ Return Payment Up to $40. │ │ │ │ │ Return Check None │ │ │ │ │ Note: This account may not be eligible for balance transfers. │ │ │ │ │ Loss of Intro APR: We will end your introductory APR if any required Minimum Payment is 60 days late, │ │ │ │ │ and │ │ │ │ │ apply the Penalty APR. │ │ │ │ │ How We Will Calculate Your Balance: We use the daily balance method (including new transactions). │ │ │ │ │ Prime Rate: Variable APRs are based on the 7.75% Prime Rate as of 2/7/2023. │ │ │ │ │ aWe add 11.74% to the Prime Rate to determine the Purchase/My Chase Loan/Balance Transfer APR. │ │ │ │ │ Maximum APR 29.99%. │ │ │ │ │ bWe add 21.74% to the Prime Rate to determine the Cash Advance APR. Maximum APR 29.99%. │ │ │ │ │ cWe add up to 26.99% to the Prime Rate to determine the Penalty APR. Maximum APR 29.99%. │ │ │ │ │ dMy Chase Plan Fee: The My Chase Plan Fee is calculated at the time each plan is created and is based │ │ │ │ │ on │ │ │ │ │ the amount of each purchase transaction or amount selected to create the plan, the number of billing │ │ │ │ │ periods │ │ │ │ │ you choose to pay the balance in full, and other factors. The monthly and aggregate dollar amount of │ │ │ │ │ your My │ │ │ │ │ Chase Plan Fee will be disclosed during the activation of each My Chase Plan. │ │ │ │ │ MILITARY LENDING ACT NOTICE: Federal law provides important protections to members of the Armed │ │ │ │ │ Forces and their dependents relating to extensions of consumer credit. In general, the cost of consumer │ │ │ │ │ credit │ │ │ │ │ to a member of the Armed Forces and his or her dependent may not exceed an annual percentage rate of 36 │ │ │ │ │ percent. This rate must include, as applicable to the credit transaction or account: the costs │ │ │ │ │ associated with │ │ │ │ │ credit insurance premiums; fees for ancillary products sold in connection with the credit transaction; │ │ │ │ │ any │ │ │ │ │ application fee charged (other than certain application fees for specified credit transactions or │ │ │ │ │ accounts); and │ │ │ │ │ any participation fee charged (other than certain participation fees for a credit card account). To │ │ │ │ │ receive this │ │ │ │ │ information and a description of your payment obligation verbally, please call 1-800-235-9978. │ │ │ │ │ TERMS & CONDITIONS │ │ │ │ │ Authorization: When you respond to this credit card offer from JPMorgan Chase Bank, N.A., Member FDIC, │ │ │ │ │ a │ │ │ │ │ subsidiary of JPMorgan Chase & Co. ("Chase", "we", or "us"), you agree to the following: │ │ │ │ │ 1. You authorize us to obtain credit bureau reports, employment, and income information about you that │ │ │ │ │ we │ │ │ │ │ will use when considering your application for credit. We may obtain and use information about your │ │ │ │ │ accounts with us and others such as Checking, Deposit, Investment, and Utility accounts from credit │ │ │ │ │ bureaus and other entities. You also authorize us to obtain credit bureau reports and any other │ │ │ │ │ information about you in connection with: 1) extensions of credit on your account; 2) the │ │ │ │ │ administration, │ │ │ │ │ review or collection of your account; and 3) offering you enhanced or additional products and services. │ │ │ │ │ If │ │ │ │ │ you ask, we will tell you the name and address of the credit bureau from which we obtained a report │ │ │ │ │ about you. │ │ │ │ │ 2. If an account is opened, you will receive a Cardmember Agreement with your card(s). You agree to the │ │ │ │ │ terms of this agreement by: using the account or any card, authorizing their use, or making any payment │ │ │ │ │ on the account. │ │ │ │ │ 3. By providing your mobile ph │ │ │ │ │ │ │ │ │ │ Extract information from this document and return a JSON that follows the correct schema. │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ Given below is XML that describes the information to extract from this document and the tags to extract │ │ │ │ │ it into. │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ <output> │ │ │ │ │ <list name="fees" description="What fees and charges are associated with my account?"> │ │ │ │ │ <object> │ │ │ │ │ <integer name="index" format="1-indexed"/> │ │ │ │ │ <string name="name" format="lower-case; two-words"/> │ │ │ │ │ <string name="explanation" format="one-line"/> │ │ │ │ │ <float name="value" format="percentage"/> │ │ │ │ │ </object> │ │ │ │ │ </list> │ │ │ │ │ <object name="interest_rates" description="What are the interest rates offered by the bank on │ │ │ │ │ savings and checking accounts, loans, and credit products?"/> │ │ │ │ │ </output> │ │ │ │ │ │ │ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │ │ │ ╭───────────────────────────────────────────── Instructions ──────────────────────────────────────────────╮ │ │ │ │ You are a helpful assistant only capable of communicating with valid JSON, and no other text. │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ ONLY return a valid JSON object (no other text is necessary), where the key of the field in JSON is the │ │ │ │ │ `name` attribute of the corresponding XML, and the value is of the type specified by the corresponding │ │ │ │ │ XML's tag. The JSON MUST conform to the XML format, including any types and format requests e.g. │ │ │ │ │ requests for lists, objects and specific types. Be correct and concise. If you are unsure anywhere, │ │ │ │ │ enter `null`. │ │ │ │ │ │ │ │ │ │ Here are examples of simple (XML, JSON) pairs that show the expected behavior: │ │ │ │ │ - `<string name='foo' format='two-words lower-case' />` => `{'foo': 'example one'}` │ │ │ │ │ - `<list name='bar'><string format='upper-case' /></list>` => `{"bar": ['STRING ONE', 'STRING TWO', │ │ │ │ │ etc.]}` │ │ │ │ │ - `<object name='baz'><string name="foo" format="capitalize two-words" /><integer name="index" │ │ │ │ │ format="1-indexed" /></object>` => `{'baz': {'foo': 'Some String', 'index': 1}}` │ │ │ │ │ │ │ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │ │ │ ╭──────────────────────────────────────────── Message History ────────────────────────────────────────────╮ │ │ │ │ ┏━━━━━━┳━━━━━━━━━┓ │ │ │ │ │ ┃ Role ┃ Content ┃ │ │ │ │ │ ┡━━━━━━╇━━━━━━━━━┩ │ │ │ │ │ └──────┴─────────┘ │ │ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │ │ │ ╭──────────────────────────────────────────── Raw LLM Output ─────────────────────────────────────────────╮ │ │ │ │ { │ │ │ │ │ "fees": [ │ │ │ │ │ { │ │ │ │ │ "index": 1, │ │ │ │ │ "name": "annual membership fee", │ │ │ │ │ "explanation": "None", │ │ │ │ │ "value": 0 │ │ │ │ │ }, │ │ │ │ │ { │ │ │ │ │ "index": 2, │ │ │ │ │ "name": "my chase plan fee", │ │ │ │ │ "explanation": "Monthly fee of 0% of the amount of each eligible purchase transaction or amount │ │ │ │ │ selected to create a My Chase Plan while in the 0% Intro Purchase APR period.\nAfter that, monthly fee │ │ │ │ │ of 1.72% of the amount of each eligible purchase transaction or amount selected to create a My Chase │ │ │ │ │ Plan. The My Chase Plan Fee will be determined at the time each My Chase Plan is created and will │ │ │ │ │ remain the same until the My Chase Plan is paid in full.", │ │ │ │ │ "value": 1.72 │ │ │ │ │ }, │ │ │ │ │ { │ │ │ │ │ "index": 3, │ │ │ │ │ "name": "balance transfers intro fee", │ │ │ │ │ "explanation": "Either $5 or 3% of the amount of each transfer, whichever is greater, on │ │ │ │ │ transfers made within 60 days of account opening. After that: Either $5 or 5% of the amount of each │ │ │ │ │ transfer, whichever is greater.", │ │ │ │ │ "value": 5 │ │ │ │ │ }, │ │ │ │ │ { │ │ │ │ │ "index": 4, │ │ │ │ │ "name": "cash advances", │ │ │ │ │ "explanation": "Either $10 or 5% of the amount of each transaction, whichever is greater.", │ │ │ │ │ "value": 5 │ │ │ │ │ }, │ │ │ │ │ { │ │ │ │ │ "index": 5, │ │ │ │ │ "name": "foreign transactions", │ │ │ │ │ "explanation": "3% of the amount of each transaction in U.S. dollars.", │ │ │ │ │ "value": 3 │ │ │ │ │ }, │ │ │ │ │ { │ │ │ │ │ "index": 6, │ │ │ │ │ "name": "late payment", │ │ │ │ │ "explanation": "Up to $40.", │ │ │ │ │ "value": 40 │ │ │ │ │ }, │ │ │ │ │ { │ │ │ │ │ "index": 7, │ │ │ │ │ "name": "over-the-credit-limit", │ │ │ │ │ "explanation": "None", │ │ │ │ │ "value": null │ │ │ │ │ }, │ │ │ │ │ { │ │ │ │ │ "index": 8, │ │ │ │ │ "name": "return payment", │ │ │ │ │ "explanation": "Up to $40.", │ │ │ │ │ "value": 40 │ │ │ │ │ }, │ │ │ │ │ { │ │ │ │ │ "index": 9, │ │ │ │ │ "name": "return check", │ │ │ │ │ "explanation": "None", │ │ │ │ │ "value": null │ │ │ │ │ } │ │ │ │ │ ], │ │ │ │ │ "interest_rates": null │ │ │ │ │ } │ │ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │ │ │ ╭─────────────────────────────────────────── Validated Output ────────────────────────────────────────────╮ │ │ │ │ SkeletonReAsk( │ │ │ │ │ incorrect_value={ │ │ │ │ │ 'fees': [ │ │ │ │ │ { │ │ │ │ │ 'index': 1, │ │ │ │ │ 'name': 'annual membership fee', │ │ │ │ │ 'explanation': 'None', │ │ │ │ │ 'value': 0.0 │ │ │ │ │ }, │ │ │ │ │ { │ │ │ │ │ 'index': 2, │ │ │ │ │ 'name': 'my chase plan fee', │ │ │ │ │ 'explanation': 'Monthly fee of 0% of the amount of each eligible purchase transaction │ │ │ │ │ or amount selected to create a My Chase Plan while in the 0% Intro Purchase APR period.\nAfter that, │ │ │ │ │ monthly fee of 1.72% of the amount of each eligible purchase transaction or amount selected to create a │ │ │ │ │ My Chase Plan. The My Chase Plan Fee will be determined at the time each My Chase Plan is created and │ │ │ │ │ will remain the same until the My Chase Plan is paid in full.', │ │ │ │ │ 'value': 1.72 │ │ │ │ │ }, │ │ │ │ │ { │ │ │ │ │ 'index': 3, │ │ │ │ │ 'name': 'balance transfers intro fee', │ │ │ │ │ 'explanation': 'Either $5 or 3% of the amount of each transfer, whichever is greater, │ │ │ │ │ on transfers made within 60 days of account opening. After that: Either $5 or 5% of the amount of each │ │ │ │ │ transfer, whichever is greater.', │ │ │ │ │ 'value': 5.0 │ │ │ │ │ }, │ │ │ │ │ { │ │ │ │ │ 'index': 4, │ │ │ │ │ 'name': 'cash advances', │ │ │ │ │ 'explanation': 'Either $10 or 5% of the amount of each transaction, whichever is │ │ │ │ │ greater.', │ │ │ │ │ 'value': 5.0 │ │ │ │ │ }, │ │ │ │ │ { │ │ │ │ │ 'index': 5, │ │ │ │ │ 'name': 'foreign transactions', │ │ │ │ │ 'explanation': '3% of the amount of each transaction in U.S. dollars.', │ │ │ │ │ 'value': 3.0 │ │ │ │ │ }, │ │ │ │ │ { │ │ │ │ │ 'index': 6, │ │ │ │ │ 'name': 'late payment', │ │ │ │ │ 'explanation': 'Up to $40.', │ │ │ │ │ 'value': 40.0 │ │ │ │ │ }, │ │ │ │ │ { │ │ │ │ │ 'index': 7, │ │ │ │ │ 'name': 'over-the-credit-limit', │ │ │ │ │ 'explanation': 'None', │ │ │ │ │ 'value': None │ │ │ │ │ }, │ │ │ │ │ { │ │ │ │ │ 'index': 8, │ │ │ │ │ 'name': 'return payment', │ │ │ │ │ 'explanation': 'Up to $40.', │ │ │ │ │ 'value': 40 │ │ │ │ │ }, │ │ │ │ │ { │ │ │ │ │ 'index': 9, │ │ │ │ │ 'name': 'return check', │ │ │ │ │ 'explanation': 'None', │ │ │ │ │ 'value': None │ │ │ │ │ } │ │ │ │ │ ], │ │ │ │ │ 'interest_rates': None │ │ │ │ │ }, │ │ │ │ │ fail_results=[ │ │ │ │ │ FailResult( │ │ │ │ │ outcome='fail', │ │ │ │ │ metadata=None, │ │ │ │ │ error_message='JSON does not match schema', │ │ │ │ │ fix_value=None │ │ │ │ │ ) │ │ │ │ │ ] │ │ │ │ │ ) │ │ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ └── ╭────────────────────────────────────────────────── Step 1 ───────────────────────────────────────────────────╮ │ ╭──────────────────────────────────────────────── Prompt ─────────────────────────────────────────────────╮ │ │ │ │ │ │ │ I was given the following JSON response, which had problems due to incorrect values. │ │ │ │ │ │ │ │ { │ │ │ │ "incorrect_value": { │ │ │ │ "fees": [ │ │ │ │ { │ │ │ │ "index": 1, │ │ │ │ "name": "annual membership fee", │ │ │ │ "explanation": "None", │ │ │ │ "value": 0.0 │ │ │ │ }, │ │ │ │ { │ │ │ │ "index": 2, │ │ │ │ "name": "my chase plan fee", │ │ │ │ "explanation": "Monthly fee of 0% of the amount of each eligible purchase transaction or amount │ │ │ │ selected to create a My Chase Plan while in the 0% Intro Purchase APR period.\nAfter that, monthly fee │ │ │ │ of 1.72% of the amount of each eligible purchase transaction or amount selected to create a My Chase │ │ │ │ Plan. The My Chase Plan Fee will be determined at the time each My Chase Plan is created and will │ │ │ │ remain the same until the My Chase Plan is paid in full.", │ │ │ │ "value": 1.72 │ │ │ │ }, │ │ │ │ { │ │ │ │ "index": 3, │ │ │ │ "name": "balance transfers intro fee", │ │ │ │ "explanation": "Either $5 or 3% of the amount of each transfer, whichever is greater, on │ │ │ │ transfers made within 60 days of account opening. After that: Either $5 or 5% of the amount of each │ │ │ │ transfer, whichever is greater.", │ │ │ │ "value": 5.0 │ │ │ │ }, │ │ │ │ { │ │ │ │ "index": 4, │ │ │ │ "name": "cash advances", │ │ │ │ "explanation": "Either $10 or 5% of the amount of each transaction, whichever is greater.", │ │ │ │ "value": 5.0 │ │ │ │ }, │ │ │ │ { │ │ │ │ "index": 5, │ │ │ │ "name": "foreign transactions", │ │ │ │ "explanation": "3% of the amount of each transaction in U.S. dollars.", │ │ │ │ "value": 3.0 │ │ │ │ }, │ │ │ │ { │ │ │ │ "index": 6, │ │ │ │ "name": "late payment", │ │ │ │ "explanation": "Up to $40.", │ │ │ │ "value": 40.0 │ │ │ │ }, │ │ │ │ { │ │ │ │ "index": 7, │ │ │ │ "name": "over-the-credit-limit", │ │ │ │ "explanation": "None", │ │ │ │ "value": null │ │ │ │ }, │ │ │ │ { │ │ │ │ "index": 8, │ │ │ │ "name": "return payment", │ │ │ │ "explanation": "Up to $40.", │ │ │ │ "value": 40 │ │ │ │ }, │ │ │ │ { │ │ │ │ "index": 9, │ │ │ │ "name": "return check", │ │ │ │ "explanation": "None", │ │ │ │ "value": null │ │ │ │ } │ │ │ │ ], │ │ │ │ "interest_rates": null │ │ │ │ }, │ │ │ │ "error_messages": [ │ │ │ │ "JSON does not match schema" │ │ │ │ ] │ │ │ │ } │ │ │ │ │ │ │ │ Help me correct the incorrect values based on the given error messages. │ │ │ │ │ │ │ │ Given below is XML that describes the information to extract from this document and the tags to extract │ │ │ │ it into. │ │ │ │ │ │ │ │ <output> │ │ │ │ <list name="fees" description="What fees and charges are associated with my account?"> │ │ │ │ <object> │ │ │ │ <integer name="index" format="1-indexed"/> │ │ │ │ <string name="name" format="lower-case; two-words"/> │ │ │ │ <string name="explanation" format="one-line"/> │ │ │ │ <float name="value" format="percentage"/> │ │ │ │ </object> │ │ │ │ </list> │ │ │ │ <object name="interest_rates" description="What are the interest rates offered by the bank on │ │ │ │ savings and checking accounts, loans, and credit products?"/> │ │ │ │ </output> │ │ │ │ │ │ │ │ │ │ │ │ ONLY return a valid JSON object (no other text is necessary), where the key of the field in JSON is the │ │ │ │ `name` attribute of the corresponding XML, and the value is of the type specified by the corresponding │ │ │ │ XML's tag. The JSON MUST conform to the XML format, including any types and format requests e.g. │ │ │ │ requests for lists, objects and specific types. Be correct and concise. If you are unsure anywhere, │ │ │ │ enter `null`. │ │ │ │ │ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │ │ ╭───────────────────────────────────────────── Instructions ──────────────────────────────────────────────╮ │ │ │ │ │ │ │ You are a helpful assistant only capable of communicating with valid JSON, and no other text. │ │ │ │ │ │ │ │ ONLY return a valid JSON object (no other text is necessary), where the key of the field in JSON is the │ │ │ │ `name` attribute of the corresponding XML, and the value is of the type specified by the corresponding │ │ │ │ XML's tag. The JSON MUST conform to the XML format, including any types and format requests e.g. │ │ │ │ requests for lists, objects and specific types. Be correct and concise. If you are unsure anywhere, │ │ │ │ enter `null`. │ │ │ │ │ │ │ │ Here are examples of simple (XML, JSON) pairs that show the expected behavior: │ │ │ │ - `<string name='foo' format='two-words lower-case' />` => `{'foo': 'example one'}` │ │ │ │ - `<list name='bar'><string format='upper-case' /></list>` => `{"bar": ['STRING ONE', 'STRING TWO', │ │ │ │ etc.]}` │ │ │ │ - `<object name='baz'><string name="foo" format="capitalize two-words" /><integer name="index" │ │ │ │ format="1-indexed" /></object>` => `{'baz': {'foo': 'Some String', 'index': 1}}` │ │ │ │ │ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │ │ ╭──────────────────────────────────────────── Message History ────────────────────────────────────────────╮ │ │ │ No message history. │ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │ │ ╭──────────────────────────────────────────── Raw LLM Output ─────────────────────────────────────────────╮ │ │ │ { │ │ │ │ "fees": [ │ │ │ │ { │ │ │ │ "index": 1, │ │ │ │ "name": "annual membership fee", │ │ │ │ "explanation": "None", │ │ │ │ "value": 0.0 │ │ │ │ }, │ │ │ │ { │ │ │ │ "index": 2, │ │ │ │ "name": "my chase plan fee", │ │ │ │ "explanation": "Monthly fee of 0% of the amount of each eligible purchase transaction or amount │ │ │ │ selected to create a My Chase Plan while in the 0% Intro Purchase APR period. After that, monthly fee │ │ │ │ of 1.72% of the amount of each eligible purchase transaction or amount selected to create a My Chase │ │ │ │ Plan. The My Chase Plan Fee will be determined at the time each My Chase Plan is created and will │ │ │ │ remain the same until the My Chase Plan is paid in full.", │ │ │ │ "value": 1.72 │ │ │ │ }, │ │ │ │ { │ │ │ │ "index": 3, │ │ │ │ "name": "balance transfers intro fee", │ │ │ │ "explanation": "Either $5 or 3% of the amount of each transfer, whichever is greater, on │ │ │ │ transfers made within 60 days of account opening. After that: Either $5 or 5% of the amount of each │ │ │ │ transfer, whichever is greater.", │ │ │ │ "value": 5.0 │ │ │ │ }, │ │ │ │ { │ │ │ │ "index": 4, │ │ │ │ "name": "cash advances", │ │ │ │ "explanation": "Either $10 or 5% of the amount of each transaction, whichever is greater.", │ │ │ │ "value": 5.0 │ │ │ │ }, │ │ │ │ { │ │ │ │ "index": 5, │ │ │ │ "name": "foreign transactions", │ │ │ │ "explanation": "3% of the amount of each transaction in U.S. dollars.", │ │ │ │ "value": 3.0 │ │ │ │ }, │ │ │ │ { │ │ │ │ "index": 6, │ │ │ │ "name": "late payment", │ │ │ │ "explanation": "Up to $40.", │ │ │ │ "value": 40.0 │ │ │ │ }, │ │ │ │ { │ │ │ │ "index": 7, │ │ │ │ "name": "over-the-credit-limit", │ │ │ │ "explanation": "None", │ │ │ │ "value": null │ │ │ │ }, │ │ │ │ { │ │ │ │ "index": 8, │ │ │ │ "name": "return payment", │ │ │ │ "explanation": "Up to $40.", │ │ │ │ "value": 40.0 │ │ │ │ }, │ │ │ │ { │ │ │ │ "index": 9, │ │ │ │ "name": "return check", │ │ │ │ "explanation": "None", │ │ │ │ "value": null │ │ │ │ } │ │ │ │ ], │ │ │ │ "interest_rates": null │ │ │ │ } │ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │ │ ╭─────────────────────────────────────────── Validated Output ────────────────────────────────────────────╮ │ │ │ SkeletonReAsk( │ │ │ │ incorrect_value={ │ │ │ │ 'fees': [ │ │ │ │ { │ │ │ │ 'index': 1, │ │ │ │ 'name': 'annual membership fee', │ │ │ │ 'explanation': 'None', │ │ │ │ 'value': 0.0 │ │ │ │ }, │ │ │ │ { │ │ │ │ 'index': 2, │ │ │ │ 'name': 'my chase plan fee', │ │ │ │ 'explanation': 'Monthly fee of 0% of the amount of each eligible purchase transaction │ │ │ │ or amount selected to create a My Chase Plan while in the 0% Intro Purchase APR period. After that, │ │ │ │ monthly fee of 1.72% of the amount of each eligible purchase transaction or amount selected to create a │ │ │ │ My Chase Plan. The My Chase Plan Fee will be determined at the time each My Chase Plan is created and │ │ │ │ will remain the same until the My Chase Plan is paid in full.', │ │ │ │ 'value': 1.72 │ │ │ │ }, │ │ │ │ { │ │ │ │ 'index': 3, │ │ │ │ 'name': 'balance transfers intro fee', │ │ │ │ 'explanation': 'Either $5 or 3% of the amount of each transfer, whichever is greater, │ │ │ │ on transfers made within 60 days of account opening. After that: Either $5 or 5% of the amount of each │ │ │ │ transfer, whichever is greater.', │ │ │ │ 'value': 5.0 │ │ │ │ }, │ │ │ │ { │ │ │ │ 'index': 4, │ │ │ │ 'name': 'cash advances', │ │ │ │ 'explanation': 'Either $10 or 5% of the amount of each transaction, whichever is │ │ │ │ greater.', │ │ │ │ 'value': 5.0 │ │ │ │ }, │ │ │ │ { │ │ │ │ 'index': 5, │ │ │ │ 'name': 'foreign transactions', │ │ │ │ 'explanation': '3% of the amount of each transaction in U.S. dollars.', │ │ │ │ 'value': 3.0 │ │ │ │ }, │ │ │ │ { │ │ │ │ 'index': 6, │ │ │ │ 'name': 'late payment', │ │ │ │ 'explanation': 'Up to $40.', │ │ │ │ 'value': 40.0 │ │ │ │ }, │ │ │ │ { │ │ │ │ 'index': 7, │ │ │ │ 'name': 'over-the-credit-limit', │ │ │ │ 'explanation': 'None', │ │ │ │ 'value': None │ │ │ │ }, │ │ │ │ { │ │ │ │ 'index': 8, │ │ │ │ 'name': 'return payment', │ │ │ │ 'explanation': 'Up to $40.', │ │ │ │ 'value': 40.0 │ │ │ │ }, │ │ │ │ { │ │ │ │ 'index': 9, │ │ │ │ 'name': 'return check', │ │ │ │ 'explanation': 'None', │ │ │ │ 'value': None │ │ │ │ } │ │ │ │ ], │ │ │ │ 'interest_rates': None │ │ │ │ }, │ │ │ │ fail_results=[ │ │ │ │ FailResult( │ │ │ │ outcome='fail', │ │ │ │ metadata=None, │ │ │ │ error_message='JSON does not match schema', │ │ │ │ fix_value=None │ │ │ │ ) │ │ │ │ ] │ │ │ │ ) │ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────╯