Skip to content

Schema Deduplication

Demonstrates schema deduplication strategies when merging OpenAPI specifications using the joiner package.

What You'll Learn

  • How to identify structurally identical schemas across documents
  • Using deduplicate-equivalent strategy for same-named schema collisions
  • Using semantic-deduplication for different-named equivalent schemas
  • Understanding when each approach applies

Prerequisites

  • Go 1.25+

Quick Start

cd examples/workflows/schema-deduplication
go run main.go

Expected Output

Schema Deduplication Strategies
================================

Scenario: Both APIs have error schemas with IDENTICAL structure
  - users-api.yaml: UserError {code, message, details}
  - products-api.yaml: ProductError {code, message, details}

These are structurally equivalent but have different names.

[1/3] Baseline: No deduplication
-----------------------------------------
  Result: Success
  Schemas in merged doc: [Product ProductError User UserError]

  Note: Both UserError and ProductError exist in output.
  This is wasteful since they're structurally identical!

[2/3] Strategy: deduplicate-equivalent
-----------------------------------------
  This strategy handles SAME-named collisions.
  When two schemas named 'Error' collide:
    - If structurally equivalent -> keep one
    - If different -> fail

  Configuration:
    SchemaStrategy: deduplicate
    EquivalenceMode: deep

  Use case: When teams independently define the same schema
  with the same name - common with shared types like Error.

[3/3] Strategy: semantic-deduplication
-----------------------------------------
  Result: Success
  Schemas in merged doc: [Product ProductError User]

  UserError was deduplicated to ProductError
     (ProductError < UserError alphabetically)

  Warnings:
    - semantic deduplication: consolidated 1 duplicate schema(s)

  Configuration:
    SemanticDeduplication: true

  The joiner identified that UserError = ProductError and consolidated them,
  rewriting the $refs that named UserError.
  See WithDeduplicationMode to keep that name rather than remove it.

=========================================
Key Takeaway:
  - deduplicate-equivalent: Merges SAME-named schemas if equivalent
  - semantic-deduplication: Finds DIFFERENT-named equivalent schemas
                            and consolidates to canonical name

Files

File Purpose
main.go Demonstrates the schema deduplication workflow
specs/users-api.yaml User service spec with User and UserError schemas
specs/products-api.yaml Product service spec with Product and ProductError schemas

Key Concepts

Deduplication Strategies

Strategy When to Use
deduplicate-equivalent Same-named schemas colliding (e.g., both have Error)
semantic-deduplication Different-named but identical schemas (e.g., UserError vs ProductError)

How deduplicate-equivalent Works

This is a collision strategy (SchemaStrategy) that triggers when two schemas have the same name:

  1. When UserAPI and ProductAPI both define Error
  2. The joiner compares their structure using EquivalenceMode (shallow or deep)
  3. If equivalent -> keeps one, discards the duplicate
  4. If different -> fails with collision error
config.SchemaStrategy = joiner.StrategyDeduplicateEquivalent
config.EquivalenceMode = "deep"

How Semantic Deduplication Works

This is a post-merge optimization that finds schemas with different names but identical structure:

  1. After merging, scans all schemas
  2. Groups schemas by structural equivalence
  3. Holds apart names that one schema tree references (e.g., if Shipment references both OriginAddress and DestinationAddress, both persist)
  4. For remaining duplicates: selects canonical name (the first one alphabetically)
  5. Under the default remove mode: removes duplicates and rewrites all $ref pointers
config.SemanticDeduplication = true

Exception to step 4: if a collision rename has already invented a name such as Api_Address, the Address your documents declared wins even though Api_Address sorts first. See Which Name Survives.

Exception to step 5: DeduplicationMode decides what becomes of a folded name. Under pointer it is kept as a $ref to the survivor rather than removed, so a consumer naming it still resolves. Reach for it when the joined document's schema names are part of a published contract: by default UserError above simply disappears. See What Becomes of the Other Names.

Equivalence Modes

Mode Comparison Depth
none Disabled (no equivalence checking)
shallow Top-level properties only
deep Full recursive comparison including nested schemas

Use Cases

  • Shared error schemas - Multiple microservices define identical error responses
  • Common data types - Teams independently define equivalent Address, Money, or Timestamp schemas
  • API consolidation - Merging specs that evolved separately but converged on structure
  • Spec cleanup - Reducing schema bloat in merged specifications

Next Steps

Generated for oastools