Skip to main content

Epistemic Validation

The Epistemic field in the schema definition is crucial for ensuring high-quality completions through epistemic validation. When enabled, this feature activates internal quality evaluation mechanisms to assess the generated completions for validity and accuracy. The system supports two distinct validation approaches: LLM-as-Judge and K-Nearest (K-Mean) embedding-based validation.

Validation Methods​

LLM-as-Judge​

The LLM-as-Judge approach uses independent language model evaluators to assess completion quality:

  • Activation: Epistemic validation is activated when the Epistemic field in a definition has Active set to true
  • Number of Judges: The Judges field specifies how many independent evaluators will assess the generated content (e.g., setting Judges: 3 will use 3 judges)
  • Evaluation Process: Internal quality judges review the completion based on specific criteria outlined in the prompt and schema. They score the completion on two dimensions:
    • Completeness: How thoroughly the completion addresses all aspects of the prompt (0-100)
    • Correctness: The accuracy and validity of the information provided (0-100)
    • The final score is calculated as the average of both dimensions
  • Best Selection: The completion with the highest average score across all judges is selected as the final result

K-Nearest (K-Mean) Embedding Validation​

The K-Nearest approach uses semantic embeddings to find the most representative completion:

  • Embedding Generation: Each completion is converted into a high-dimensional embedding vector that captures its semantic meaning
  • Average Calculation: The system calculates the average (centroid) of all completion embeddings
  • Distance Measurement: Using Euclidean distance, the system measures how close each completion is to the average embedding
  • Best Selection: The completion whose embedding is closest to the average is selected, representing the most consensus-aligned response
  • Confidence Scoring: Results are scored based on their distance from the average, with closer results receiving higher confidence scores (normalized to 0-1 range, then scaled to 0-100)

This method is particularly effective for:

  • Finding consensus among multiple generation attempts
  • Reducing outlier or anomalous completions
  • Semantic consistency validation without requiring explicit criteria

Configuration​

LLM-as-Judge Configuration​

Definition {
    Type: String,
    Instruction: "Generate a factual summary",
    Epistemic: EpistemicValidation{
        Active: true,
        Judges: 3, // Use 3 independent judges
    },
}

K-Mean Embedding Configuration​

The K-Mean approach automatically activates when epistemic validation is enabled and uses the configured embedding model:

Definition {
    Type: String,
    Instruction: "Generate a factual summary",
    Epistemic: EpistemicValidation{
        Active: true,
        Judges: 3, // Number of completions to generate and compare
    },
    Properties: map[string]Definition{
        "embedding": {
            Type: Vector,
            Model: "text-embedding-3-small", // Specify embedding model
        },
    },
}

Benefits of Using Epistemic Validation​

General Benefits​

  • High-Quality Output: Ensures that the generated content is of the highest quality by filtering through multiple independent evaluations
  • Reliability: Reduces the chances of receiving inaccurate or invalid completions, providing more reliable results for developers
  • Efficiency: Saves developers time by automatically vetting completions through multiple judges, allowing them to focus on integrating and utilizing the best outputs
  • Configurable Rigor: Adjust the number of judges based on your quality requirements - more judges provide higher confidence but at increased cost
  • Transparency: All evaluated choices are stored in the metadata with their scores and confidence levels, enabling audit and analysis

LLM-as-Judge Benefits​

  • Criteria-Based Evaluation: Explicitly evaluates completions on completeness and correctness
  • Interpretable Scores: Provides clear numeric scores (0-100) for each evaluation dimension
  • Flexible Assessment: Can adapt evaluation criteria based on the prompt and schema requirements

K-Nearest Benefits​

  • Mathematical Objectivity: Uses geometric distance rather than subjective evaluation
  • Consensus Finding: Automatically identifies the most representative answer among multiple attempts
  • Outlier Filtering: Naturally excludes anomalous or extreme responses that deviate from the consensus
  • Semantic Consistency: Ensures the selected completion aligns with the semantic center of all generated options
  • Cost Efficiency: May be more economical than running additional LLM judge calls for certain use cases

Choosing the Right Method​

  • Use LLM-as-Judge when you need explicit evaluation criteria, interpretable scores, or want to emphasize correctness and completeness
  • Use K-Nearest when seeking consensus among multiple generations, filtering outliers, or working with semantic similarity requirements

By using the Epistemic field effectively with the appropriate validation method, developers can enhance the quality and reliability of the generated content, ensuring that only the most accurate and epistemically valid completions are utilized in their applications.