Skip to content

Cache Step Results

Avoid redundant computation by caching step results on disk, enabling faster re-runs when processing the same inputs.

QType YAML

steps:
  - type: LLMInference
    id: classify
    model: nova
    inputs: [prompt]
    outputs: [category]
    cache_config:
      namespace: document_classification  # Logical grouping for cached data
      version: "1.0"                       # Change to invalidate cache
      on_error: Drop                       # Don't cache errors (default)
      ttl: 3600                            # Cache for 1 hour (seconds)
      compress: false                      # Optionally compress cached data

Explanation

  • cache_config: Enables step-level caching with configuration options
  • namespace: Logical separation for cache entries (e.g., different projects or data domains)
  • version: Cache version string - increment to invalidate all cached results for this step
  • on_error: How to handle errors - Drop (don't cache errors, default) or Cache (cache error results)
  • ttl: Time-to-live in seconds before cached entries expire
  • compress: Whether to compress cached data (saves disk space, adds CPU overhead)

Cached values are stored in the .qtype-cache/ directory in your working directory.

Monitoring Cache Performance

Use the --progress flag to see cache hits and misses:

qtype run app.qtype.yaml --flow my_flow --progress

First run shows cache misses: 

Step classify    ✔ 5 succeeded ✖ 0 errors ⟳ 0 hits ✗ 5 misses

Subsequent runs show cache hits (much faster): 

Step classify    ✔ 5 succeeded ✖ 0 errors ⟳ 5 hits ✗ 0 misses

Complete Example

id: cache_step_demo
description: |
  Demonstrates step caching to avoid redundant computation.
  On first run, classifications are cached. On subsequent runs with same inputs,
  results are retrieved from cache.

auths:
  - type: aws
    id: aws_auth
    region: us-east-1

models:
  - type: Model
    id: nova
    provider: aws-bedrock
    model_id: us.amazon.nova-micro-v1:0
    auth: aws_auth

flows:
  - id: classify_documents
    variables:
      - id: file_path
        type: text
      - id: document
        type: text
      - id: prompt
        type: text
      - id: category
        type: text
      - id: output_file
        type: text

    inputs:
      - file_path

    outputs:
      - output_file

    steps:
      - type: FileSource
        id: load_docs
        path: file_path
        outputs: [document]

      - type: PromptTemplate
        id: create_classification_prompt
        template: |
          Classify this document into one of these categories:
          - Technology
          - Finance
          - Healthcare
          - Education

          Document: {document}

          Reply with only the category name.
        inputs: [document]
        outputs: [prompt]

      - type: LLMInference
        id: classify
        model: nova
        inputs: [prompt]
        outputs: [category]
        # Enable caching with configuration
        cache_config:
          namespace: document_classification  # Logical separation for cache
          version: "1.0"                       # Bump this to invalidate cache
          on_error: Drop                       # Don't cache errors (default)
          ttl: 3600                            # Cache for 1 hour (seconds)
          compress: false                      # Optionally compress cached data

      - type: FileWriter
        id: write_results
        path:
          uri: classification_results.parquet
        inputs: [document, category]
        outputs: [output_file]

Run the example: 

# First run - cold cache
qtype run examples/data_processing/cache_step_results.qtype.yaml --progress -i '{"file_path": "examples/data_processing/sample_documents.jsonl"}'

# Second run - warm cache (much faster)
qtype run examples/data_processing/cache_step_results.qtype.yaml  --progress -i '{"file_path": "examples/data_processing/sample_documents.jsonl"}'

See Also