Summarize text accurately
Note
To download this example as a Jupyter notebook, click here.
In this example, we will use Guardrails in the summarization of a text document. We will check whether the summarized document has a high semantic similarity with the original document.
Objective
Summarize a text document and check whether the summarized document has a high semantic similarity with the original document.
Step 0: Setup
In order to run this example, you will need to install the numpy package. You can do so by running the following commands:
Step 1: Create the RAIL Spec
Ordinarily, we would create an RAIL spec in a separate file. For the purposes of this example, we will create the spec in this notebook as a string following the RAIL syntax. For more information on RAIL, see the RAIL documentation. We will also show the same RAIL spec in a code-first format using a Pydantic model.
In this RAIL spec, we:
- Create an
outputschema that returns a single key-value pair. The key should be 'summary', and the value should be the summary of the given document.
First let's open our document:
Next we can define our RAIL spec either as a XML string:
from string import Template
rail_str = Template("""
<rail version="0.1">
<output>
<string description="Summarize the given document faithfully." format="similar-to-document: {${document}}, 0.60" name="summary" on-fail-similar-to-document="filter"></string>
</output>
<prompt>
Summarize the following document:
${document}
${gr.complete_json_suffix}
</prompt>
</rail>
""").safe_substitute(document=document)
Or as a Pydantic model:
from pydantic import BaseModel, Field
from guardrails.validators import SimilarToDocument
prompt = """
Summarize the following document:
${document}
${gr.complete_json_suffix}
"""
class DocumentSummary(BaseModel):
summary: str = Field(
description="Summarize the given document faithfully.",
validators=[SimilarToDocument(document=f"'{content}'", threshold=0.60, on_fail="filter")]
)
Note
In order to ensure the summary is similar to the document, we use similar-to-document as the validator. This validator embeds the document and the summary and checks whether the cosine similarity between the two embeddings is above a threshold.
Step 2: Create a Guard object with the RAIL Spec
We create a gd.Guard object that will check, validate and correct the output of the LLM. This object:
- Enforces the quality criteria specified in the RAIL spec.
- Takes corrective action when the quality criteria are not met.
- Compiles the schema and type info from the RAIL spec and adds it to the prompt.
From our RAIL string:
Or from our Pydantic model:
We see the prompt that will be sent to the LLM:
Here, statement_to_be_translated is the the statement and will be provided by the user at runtime.
Step 3: Wrap the LLM API call with Guard
First, let's try translating a statement that doesn't have any profanity in it.
In order to see a detailed look into the logs of the Guard object, we can print the Guard state history:
The guard wrapper returns the raw_llm_respose (which is a simple string), and the validated and corrected output (which is a dictionary). We can see that the output is a dictionary with the correct schema and types.
Next, let's try using a smaller model, which is not going to be good at summarization. We can see that the output is filtered out.
We can see the step-wise history of the Guard object below: