Epistemic Validation

Model confidence and uncertainty in generated content. Use epistemic markers to capture when the AI is uncertain and needs to express probabilistic information.

Overview

Epistemic uncertainty allows you to:

• Capture confidence levels in generated facts
• Express probabilistic information when certainty is low
• Flag uncertain data for human review
• Generate with awareness of knowledge limitations

World Building with Uncertainty Example

This example generates a fantasy world's technological landscape with epistemic uncertainty markers:

cURL

curl -X POST http://localhost:2008/objectGen \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer YOUR_API_KEY" \
     -d '{
          "prompt": "Generate a schema that defines the technological landscape of the world",
          "definition": {
            "type": "object",
            "instruction": "Defines the technological landscape of the world, including its level of advancement and notable innovations.",
            "properties": {
              "Level": {
                "type": "string",
                "instruction": "Categorize the overall technological sophistication of the world, such as medieval, industrial, or advanced futuristic.",
                "epistemic": {
                  "active": true
                }
              },
              "Inventions": {
                "type": "string",
                "instruction": "Describe the most significant technological discoveries and their transformative impact on the society, economy, and daily life.",
                "epistemic": {
                  "active": true
                }
              }
            }
          }
        }'

Python

import requests
import json

url = "http://localhost:2008/objectGen"
headers = {
    "Content-Type": "application/json",
    "Authorization": "Bearer YOUR_API_KEY"
}

data = {
    "prompt": "Generate a schema that defines the technological landscape of the world",
    "definition": {
        "type": "object",
        "instruction": "Defines the technological landscape of the world, including its level of advancement and notable innovations.",
        "properties": {
            "Level": {
                "type": "string",
                "instruction": "Categorize the overall technological sophistication of the world, such as medieval, industrial, or advanced futuristic.",
                "epistemic": {
                    "active": True
                }
            },
            "Inventions": {
                "type": "string",
                "instruction": "Describe the most significant technological discoveries and their transformative impact on the society, economy, and daily life.",
                "epistemic": {
                    "active": True
                }
            }
        }
    }
}

response = requests.post(url, headers=headers, json=data)
result = response.json()

print("Generated Content:")
print(json.dumps(result, indent=2))

# Access epistemic metadata
if "epistemic_metadata" in result:
    print("\nConfidence Levels:")
    for field, metadata in result["epistemic_metadata"].items():
        print(f"  {field}: {metadata.get('confidence', 'N/A')}")

JavaScript/Node.js

const fetch = require('node-fetch');

const url = 'http://localhost:2008/objectGen';
const headers = {
    'Content-Type': 'application/json',
    'Authorization': 'Bearer YOUR_API_KEY'
};

const data = {
    prompt: 'Generate a schema that defines the technological landscape of the world',
    definition: {
        type: 'object',
        instruction: 'Defines the technological landscape of the world, including its level of advancement and notable innovations.',
        properties: {
            Level: {
                type: 'string',
                instruction: 'Categorize the overall technological sophistication of the world, such as medieval, industrial, or advanced futuristic.',
                epistemic: {
                    active: true
                }
            },
            Inventions: {
                type: 'string',
                instruction: 'Describe the most significant technological discoveries and their transformative impact on the society, economy, and daily life.',
                epistemic: {
                    active: true
                }
            }
        }
    }
};

fetch(url, {
    method: 'POST',
    headers: headers,
    body: JSON.stringify(data)
})
.then(response => response.json())
.then(result => {
    console.log('Generated Content:');
    console.log(JSON.stringify(result, null, 2));
    
    if (result.epistemic_metadata) {
        console.log('\nConfidence Levels:');
        Object.entries(result.epistemic_metadata).forEach(([field, metadata]) => {
            console.log(`  ${field}: ${metadata.confidence || 'N/A'}`);
        });
    }
})
.catch(error => console.error('Error:', error));

Go

package main

import (
    "bytes"
    "encoding/json"
    "fmt"
    "io/ioutil"
    "net/http"
)

type Epistemic struct {
    Active bool `json:"active"`
}

type Property struct {
    Type        string     `json:"type"`
    Instruction string     `json:"instruction"`
    Epistemic   *Epistemic `json:"epistemic,omitempty"`
}

type Definition struct {
    Type        string              `json:"type"`
    Instruction string              `json:"instruction"`
    Properties  map[string]Property `json:"properties"`
}

type RequestBody struct {
    Prompt     string      `json:"prompt"`
    Definition *Definition `json:"definition"`
}

type EpistemicMetadata struct {
    Confidence float64 `json:"confidence"`
    Reasoning  string  `json:"reasoning"`
}

type Response struct {
    Level              string                       `json:"Level"`
    Inventions         string                       `json:"Inventions"`
    EpistemicMetadata  map[string]EpistemicMetadata `json:"epistemic_metadata"`
}

func main() {
    url := "http://localhost:2008/objectGen"
    
    requestBody := RequestBody{
        Prompt: "Generate a schema that defines the technological landscape of the world",
        Definition: &Definition{
            Type:        "object",
            Instruction: "Defines the technological landscape of the world, including its level of advancement and notable innovations.",
            Properties: map[string]Property{
                "Level": {
                    Type:        "string",
                    Instruction: "Categorize the overall technological sophistication of the world, such as medieval, industrial, or advanced futuristic.",
                    Epistemic:   &Epistemic{Active: true},
                },
                "Inventions": {
                    Type:        "string",
                    Instruction: "Describe the most significant technological discoveries and their transformative impact on the society, economy, and daily life.",
                    Epistemic:   &Epistemic{Active: true},
                },
            },
        },
    }
    
    jsonData, err := json.Marshal(requestBody)
    if err != nil {
        fmt.Printf("Error marshalling request: %v\n", err)
        return
    }
    
    req, err := http.NewRequest("POST", url, bytes.NewBuffer(jsonData))
    if err != nil {
        fmt.Printf("Error creating request: %v\n", err)
        return
    }
    
    req.Header.Set("Content-Type", "application/json")
    req.Header.Set("Authorization", "Bearer YOUR_API_KEY")
    
    client := &http.Client{}
    resp, err := client.Do(req)
    if err != nil {
        fmt.Printf("Error sending request: %v\n", err)
        return
    }
    defer resp.Body.Close()
    
    body, err := ioutil.ReadAll(resp.Body)
    if err != nil {
        fmt.Printf("Error reading response: %v\n", err)
        return
    }
    
    var result Response
    if err := json.Unmarshal(body, &result); err != nil {
        fmt.Printf("Error unmarshalling response: %v\n", err)
        return
    }
    
    fmt.Println("Generated Content:")
    fmt.Printf("Level: %s\n", result.Level)
    fmt.Printf("Inventions: %s\n", result.Inventions)
    
    if len(result.EpistemicMetadata) > 0 {
        fmt.Println("\nConfidence Levels:")
        for field, metadata := range result.EpistemicMetadata {
            fmt.Printf("  %s: %.2f - %s\n", field, metadata.Confidence, metadata.Reasoning)
        }
    }
}

Expected Response

{
  "Level": "Advanced Industrial Revolution with Early Electrical Age characteristics",
  "Inventions": "Steam-powered mechanization and electrical telegraphy have fundamentally transformed manufacturing efficiency and long-distance communication, enabling rapid urbanization and global trade networks.",
  "epistemic_metadata": {
    "Level": {
      "confidence": 0.75,
      "reasoning": "The categorization is based on typical technological markers, but the specific world context may have unique factors that affect this classification.",
      "alternatives": [
        "Late Industrial with Experimental Electrical",
        "Proto-Modern Transitional Era"
      ]
    },
    "Inventions": {
      "confidence": 0.68,
      "reasoning": "While these are historically significant innovations, without specific context about this particular world's history, there's uncertainty about which inventions had the most transformative impact.",
      "caveats": [
        "Impact may vary based on cultural adoption rates",
        "Regional differences in technological distribution not accounted for"
      ]
    }
  }
}

Epistemic Configuration

Basic Epistemic Flag

Enable epistemic uncertainty tracking for a field:

{
  "type": "string",
  "instruction": "Your instruction here",
  "epistemic": {
    "active": true
  }
}

Advanced Epistemic Options

{
  "type": "string",
  "instruction": "Your instruction here",
  "epistemic": {
    "active": true,
    "confidence_threshold": 0.7,
    "require_alternatives": true,
    "require_reasoning": true
  }
}

Epistemic Metadata Structure

When epistemic uncertainty is active, the response includes metadata:

Field	Type	Description
confidence	number	Confidence level (0.0 - 1.0)
reasoning	string	Explanation of the confidence level
alternatives	array	Alternative possible values (if applicable)
caveats	array	Important qualifications or limitations
sources	array	Knowledge sources referenced (if applicable)

Use Cases

1. Factual Content Generation: Flag uncertain facts for verification
2. Research Summaries: Express confidence in synthesized information
3. Prediction Tasks: Model uncertainty in forecasts
4. Knowledge Extraction: Identify areas where more information is needed
5. Content Moderation: Flag content that may need review

Practical Example: Scientific Paper Summary

{
  "prompt": "Summarize the key findings of a hypothetical research paper on quantum computing",
  "definition": {
    "type": "object",
    "properties": {
      "main_finding": {
        "type": "string",
        "instruction": "State the primary conclusion of the research",
        "epistemic": {
          "active": true,
          "require_alternatives": true
        }
      },
      "methodology": {
        "type": "string",
        "instruction": "Describe the research methodology used",
        "epistemic": {
          "active": true
        }
      },
      "impact_assessment": {
        "type": "string",
        "instruction": "Evaluate the potential impact on the field",
        "epistemic": {
          "active": true,
          "confidence_threshold": 0.6
        }
      }
    }
  }
}

Best Practices

1. Use Selectively: Enable epistemic tracking only for fields where uncertainty matters
2. Set Thresholds: Use confidence_threshold to filter low-confidence outputs
3. Request Alternatives: Use require_alternatives for multi-option scenarios
4. Combine with Decision Points: Route low-confidence outputs for refinement
5. Document Usage: Track which fields commonly have low confidence

Confidence Interpretation Guide

Confidence Range	Interpretation	Action
0.9 - 1.0	Very High	Accept as-is
0.7 - 0.89	High	Use with minor review
0.5 - 0.69	Medium	Review and verify
0.3 - 0.49	Low	Manual verification required
0.0 - 0.29	Very Low	Regenerate or research

Combining Features

Combine epistemic uncertainty with decision points to automatically route low-confidence outputs for regeneration or human review.

Next Steps

• See Decision Trees to route based on confidence
• Explore Code Assessment for code review patterns
• Learn about Basic Examples for simple object generation