Elasticsearch Integration

📤 Direct Posting

📁 Flat NDJSON Integration

🌐 Data Schema & Queries

Direct Posting to Elasticsearch

Native Elasticsearch Integration

TYCHON Quantum Readiness provides native Elasticsearch integration through the Bulk API. The scanner posts results directly to Elasticsearch with automatic data type separation, real-time ingestion, and ECS-compliant field mapping.

🚀 Real-Time Ingestion

Results posted immediately as scanner discovers assets

🔍 Automatic Routing

Data automatically separated into specialized indices

✅ Pre-Scan Validation

Connectivity tested before scans begin

How It Works

1️⃣

Scanner discovers crypto assets

TLS connections, certificates, keystores, SSH keys, processes

2️⃣

Results sent via Bulk API

Documents batched and posted to Elasticsearch

3️⃣

Automatic index routing

Scanner determines document type and routes to appropriate index

4️⃣

Data ready for dashboard

Visualizations update in real-time as data arrives

Command-Line Switches

TYCHON provides native Elasticsearch integration through these command-line switches:

Switch	Description	Example	Full	Standard	Minimal
-posttoelastic	Enable posting scan results to Elasticsearch with connectivity test	-posttoelastic	✓	✓	✗
-elasticnode	Elasticsearch cluster URL (required)	-elasticnode "https://elastic.company.com:9200"	✓	✓	✗
-elasticapikey	Elasticsearch API key for authentication (optional for open clusters)	-elasticapikey "your-api-key"	✓	✓	✗
-elasticindex	Custom suffix for data type separation indexes	-elasticindex "mysite"	✓	✓	✗
-insecure	Skip SSL certificate verification for HTTPS connections	-insecure	✓	✓	✗

🆕 Data Type Separation with -elasticindex

The -elasticindex parameter creates multiple specialized indexes with your custom suffix:

Input: -elasticindex "production"
Creates:
• tychon-pqc-ciphers-production
• tychon-pqc-keystores-production
• tychon-pqc-certificates-production

Input: -elasticindex "dev-team"
Creates:
• tychon-pqc-ssh-keys-dev-team
• tychon-pqc-crypto-libraries-dev-team
• tychon-pqc-processes-dev-team

Configuration

Connectivity Test

TYCHON performs automatic connectivity testing before starting scans:

✅ Pre-Scan Connectivity Validation

TYCHON tests Elasticsearch connectivity immediately after version output and before cipher intelligence loading. This provides instant feedback about configuration issues without waiting for complete scans.

SSL Certificate Verification

Production Environments: By default, TYCHON validates SSL certificates for secure Elasticsearch connections.
Development/Testing: Use the -insecure flag to skip certificate verification for self-signed or invalid certificates.

# Example: Skip SSL verification for self-signed certificates
./certscanner -posttoelastic -elasticnode "https://dev-elastic:9200" -insecure

Deployment Examples

Examples showing how to use direct posting with different scan modes:

Windows Linux macOS

# Remote scan with data type separation (Windows)
.\certscanner-windows-amd64.exe -host example.com `
  -posttoelastic `
  -elasticnode "https://elastic.company.com:9200" `
  -elasticapikey "your-api-key" `
  -elasticindex "production"
# Creates: tychon-pqc-ciphers-production, tychon-pqc-certificates-production

# Local comprehensive scan with environment suffix
.\certscanner-windows-amd64.exe -mode local `
  -scanfilesystem -scanmemory -scanconnected `
  -posttoelastic `
  -elasticnode "https://elastic.company.com:9200" `
  -elasticapikey "your-api-key" `
  -elasticindex "corp-laptops"
# Creates: tychon-pqc-keystores-corp-laptops, tychon-pqc-processes-corp-laptops

# Network discovery with custom suffix
.\certscanner-windows-amd64.exe -cidr 192.168.1.0/24 `
  -ports 443,22,8443 `
  -cipherscan `
  -posttoelastic `
  -elasticnode "https://elastic.company.com:9200" `
  -elasticapikey "your-api-key" `
  -elasticindex "dmz-scan"
# Creates: tychon-pqc-ciphers-dmz-scan, tychon-pqc-ssh-keys-dmz-scan

# Remote scan with data type separation (Linux)
./certscanner-linux-amd64 -host example.com \
  -posttoelastic \
  -elasticnode "https://elastic.company.com:9200" \
  -elasticapikey "your-api-key" \
  -elasticindex "production"
# Creates: tychon-pqc-ciphers-production, tychon-pqc-certificates-production

# Local comprehensive scan with environment suffix
./certscanner-linux-amd64 -mode local \
  -scanfilesystem -scanmemory -scanconnected \
  -posttoelastic \
  -elasticnode "https://elastic.company.com:9200" \
  -elasticapikey "your-api-key" \
  -elasticindex "servers"
# Creates: tychon-pqc-keystores-servers, tychon-pqc-processes-servers

# Network discovery with custom suffix
./certscanner-linux-amd64 -cidr 192.168.1.0/24 \
  -ports 443,22,8443 \
  -cipherscan \
  -posttoelastic \
  -elasticnode "https://elastic.company.com:9200" \
  -elasticapikey "your-api-key" \
  -elasticindex "network-scan"
# Creates: tychon-pqc-ciphers-network-scan, tychon-pqc-ssh-keys-network-scan

# Remote scan with Elasticsearch output (macOS)
# For Intel Macs:
./certscanner-darwin-amd64 -host example.com \
  -posttoelastic \
  -elasticnode "https://elastic.company.com:9200" \
  -elasticapikey "your-api-key" \
  -elasticindex "production"

# For Apple Silicon (M1/M2/M3):
./certscanner-darwin-arm64 -host example.com \
  -posttoelastic \
  -elasticnode "https://elastic.company.com:9200" \
  -elasticapikey "your-api-key" \
  -elasticindex "production"

# Local comprehensive scan
./certscanner-darwin-arm64 -mode local \
  -scanfilesystem -scanmemory -scanconnected \
  -posttoelastic \
  -elasticnode "https://elastic.company.com:9200" \
  -elasticapikey "your-api-key" \
  -elasticindex "macos-endpoints"
# Creates: tychon-pqc-keystores-macos-endpoints, tychon-pqc-processes-macos-endpoints

Troubleshooting

Connection Refused / Cannot Connect

Verify Elasticsearch URL is correct and accessible
Check if Elasticsearch is running: curl https://your-elasticsearch:9200
Verify network connectivity and firewall rules
Check if the API key is valid

SSL Certificate Errors

For self-signed certificates in development, use -insecure flag
For production, ensure certificate chain is properly configured
Verify system trust store includes CA certificates

Authentication Errors

Verify API key is correct and not expired
Check API key has permissions to create indices and write documents
Ensure API key format is correct (no quotes inside the key itself)

Data Not Appearing in Expected Indices

Verify index template is deployed (see Universal Elements tab)
Check if indices are being created: curl 'https://your-elasticsearch:9200/_cat/indices/tychon-pqc-*'
Verify custom suffix matches your -elasticindex parameter
Check scanner output for any error messages

Flat NDJSON Integration

File-Based Elasticsearch Integration

This integration method uses Custom Logs (Filestream) in Kibana Fleet to monitor NDJSON files written by the scanner. An ingest pipeline reconstructs nested objects from flat dot notation, and transforms route events to specialized indices with upsert support.

📁 File-Based

Scanner writes NDJSON files to filesystem

🔄 Nested Reconstruction

Ingest pipeline rebuilds dashboard-required nested objects

🎯 Upsert Support

Transforms use _id field for document updates

Architecture

Scanner writes flat NDJSON files to filesystem
    ↓
Custom Logs (Filestream) Integration monitors files
    ↓ (NDJSON parser)
Ingest Pipeline (tychon-pqc-flat-ndjson-pipeline)
    - Reconstructs nested objects from flat dot notation
    - Maps to ECS fields
    - Sets event routing
    ↓
Data Stream (logs-tychon-pqc-default)
    ↓
7 Transforms route by event type, upsert by _id
    ↓
Destination Indices: tychon-pqc-ciphers*, tychon-pqc-certificates*, etc.
    ↓
Existing Dashboard (no changes needed)

Key Features

✅ Nested Object Reconstruction: Transforms flat certificate.leaf.subject.common_name → tychon.certificate_leaf_details.subject.common_name
✅ Upsert Support: Uses _id field from flat NDJSON for document updates (not duplicates)
✅ Event Routing: Automatically routes events to appropriate indices based on type
✅ Dashboard Compatible: Works with existing Tychon Quantum Readiness dashboard without modifications
✅ Data Stream Cleanup: Automatically removes data from data stream after 7 days

Prerequisites

Software Requirements

✅ Elasticsearch 8.x cluster (running and accessible)
✅ Kibana 8.x (connected to Elasticsearch)
✅ Fleet Server configured and running
✅ Elastic Agent 8.x installed on scanner host(s)
✅ TYCHON PQC Scanner

Access Requirements

✅ Elasticsearch cluster admin access
✅ Kibana admin access (for Fleet configuration)
✅ SSH/RDP access to scanner host(s)
✅ Network connectivity from scanner to Elasticsearch/Kibana

Step-by-Step Deployment

Step 1: Deploy Index Template

The index template must be deployed before any data is ingested. It defines field mappings for all tychon-pqc-* and logs-tychon-pqc-* indices.

📥 Download the template: tychon-pqc-index-template.json

⚠️

Template Conflict Warning

If you previously used -posttoelastic, you may see this error:

illegal_argument_exception: index template [tychon-pqc] has index patterns
matching patterns from existing templates [tychon-pqc-template] with the same priority [200]

Solution: Delete the old template first using one of these methods:

Option A: Kibana Dev Tools

DELETE _index_template/tychon-pqc-template

Option B: Command Line

curl -X DELETE "https://your-elasticsearch:9200/_index_template/tychon-pqc-template" \
  -u "elastic:yourpassword"

Then proceed with creating the new template below. The new template supports both -posttoelastic and Filestream integration.

✓ Note: If you run -posttoelastic after creating the new template, you'll see a warning about template creation failing. This is expected and harmless - the scanner will continue posting data successfully using the new template.

Via Kibana Dev Tools

Copy the entire block below and paste it into Kibana Dev Tools console, then click the green ▶ play button:

💡 Tip: Click inside the code block below, press Ctrl+A (Cmd+A on Mac) to select all, then Ctrl+C (Cmd+C) to copy.

PUT _index_template/tychon-pqc
{
  "index_patterns": [
    "tychon-pqc-*",
    "logs-tychon-pqc-*"
  ],
  "priority": 200,
  "template": {
    "mappings": {
      "dynamic": true,
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "active": {
          "type": "boolean"
        },
        "archive": {
          "properties": {
            "encryption": {
              "properties": {
                "enabled": {
                  "type": "boolean"
                },
                "strength": {
                  "ignore_above": 256,
                  "type": "keyword"
                },
                "type": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              },
              "type": "object"
            },
            "format": {
              "properties": {
                "version": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              },
              "type": "object"
            },
            "type": {
              "ignore_above": 256,
              "type": "keyword"
            }
          },
          "type": "object"
        },
        "asset_type": {
          "ignore_above": 256,
          "type": "keyword"
        },
        "certificate": {
          "properties": {
            "algorithm": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "bit_size": {
              "type": "long"
            },
            "cert_store_locations": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "curve": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "extended_key_usage": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "has_client_cert": {
              "type": "boolean"
            },
            "is_file": {
              "type": "boolean"
            },
            "is_private_key": {
              "type": "boolean"
            },
            "is_self_signed": {
              "type": "boolean"
            },
            "issuer": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "issuer_common_name": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "key_usage": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "not_after": {
              "type": "date"
            },
            "not_before": {
              "type": "date"
            },
            "serial_number": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "subject": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "subject_common_name": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "version": {
              "type": "long"
            }
          },
          "type": "object"
        },
        "crypto": {
          "properties": {
            "curve": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "fingerprint_sha1": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "key_algorithm": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "key_size": {
              "type": "long"
            },
            "signature_algorithm": {
              "ignore_above": 256,
              "type": "keyword"
            }
          },
          "type": "object"
        },
        "error": {
          "properties": {
            "code": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "message": {
              "fields": {
                "keyword": {
                  "ignore_above": 512,
                  "type": "keyword"
                }
              },
              "type": "text"
            }
          },
          "type": "object"
        },
        "event": {
          "properties": {
            "action": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "category": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "dataset": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "duration": {
              "type": "long"
            },
            "kind": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "outcome": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "risk_score": {
              "type": "float"
            },
            "severity": {
              "type": "long"
            }
          },
          "type": "object"
        },
        "file": {
          "properties": {
            "accessed": {
              "type": "date"
            },
            "created": {
              "type": "date"
            },
            "ctime": {
              "type": "date"
            },
            "extension": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "group": {
              "properties": {
                "id": {
                  "ignore_above": 256,
                  "type": "keyword"
                },
                "name": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              },
              "type": "object"
            },
            "hash": {
              "properties": {
                "md5": {
                  "ignore_above": 256,
                  "type": "keyword"
                },
                "sha1": {
                  "ignore_above": 256,
                  "type": "keyword"
                },
                "sha1_certificate": {
                  "ignore_above": 256,
                  "type": "keyword"
                },
                "sha256": {
                  "ignore_above": 256,
                  "type": "keyword"
                },
                "sha256_certificate": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              },
              "type": "object"
            },
            "mtime": {
              "type": "date"
            },
            "name": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "owner": {
              "properties": {
                "id": {
                  "ignore_above": 256,
                  "type": "keyword"
                },
                "name": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              },
              "type": "object"
            },
            "path": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "permissions": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "size": {
              "type": "long"
            }
          },
          "type": "object"
        },
        "hash": {
          "properties": {
            "sha1": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "sha1_certificate": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "sha256": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "sha256_certificate": {
              "ignore_above": 256,
              "type": "keyword"
            }
          },
          "type": "object"
        },
        "host": {
          "properties": {
            "address": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "domain": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "hostname": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "id": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "ip": {
              "type": "ip"
            },
            "ipv4": {
              "type": "ip"
            },
            "ipv6": {
              "type": "ip"
            },
            "mac": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "name": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "type": {
              "ignore_above": 256,
              "type": "keyword"
            }
          },
          "type": "object"
        },
        "id": {
          "doc_values": true,
          "type": "keyword"
        },
        "ipsec_tunnel": {
          "properties": {
            "active": {
              "type": "boolean"
            },
            "config_path": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "detection_confidence": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "detection_method": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "first_detected": {
              "type": "date"
            },
            "implementation": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "last_seen": {
              "type": "date"
            },
            "name": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "status": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "version": {
              "ignore_above": 256,
              "type": "keyword"
            }
          },
          "type": "object"
        },
        "java": {
          "properties": {
            "manifest": {
              "dynamic": true,
              "type": "object"
            },
            "vendor": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "version": {
              "ignore_above": 256,
              "type": "keyword"
            }
          },
          "type": "object"
        },
        "keystore": {
          "properties": {
            "accessible": {
              "type": "boolean"
            },
            "cert_count": {
              "type": "long"
            },
            "error_message": {
              "fields": {
                "keyword": {
                  "ignore_above": 512,
                  "type": "keyword"
                }
              },
              "type": "text"
            },
            "owner": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "permissions": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "requires_auth": {
              "type": "boolean"
            },
            "stats": {
              "properties": {
                "certificate_types": {
                  "properties": {
                    "ca": {
                      "type": "long"
                    },
                    "end_entity": {
                      "type": "long"
                    }
                  },
                  "type": "object"
                },
                "expired_certificates": {
                  "type": "long"
                },
                "expiring_soon_certificates": {
                  "type": "long"
                },
                "key_algorithms": {
                  "dynamic": true,
                  "properties": {
                    "DSA": {
                      "type": "integer"
                    },
                    "ECDH": {
                      "type": "integer"
                    },
                    "ECDSA": {
                      "type": "integer"
                    },
                    "Ed25519": {
                      "type": "integer"
                    },
                    "RSA": {
                      "type": "integer"
                    }
                  },
                  "type": "object"
                },
                "pqc_vulnerable_certificates": {
                  "type": "long"
                },
                "vulnerable_certificates": {
                  "type": "long"
                }
              },
              "type": "object"
            },
            "type": {
              "ignore_above": 256,
              "type": "keyword"
            }
          },
          "type": "object"
        },
        "last_seen": {
          "type": "date"
        },
        "library": {
          "properties": {
            "additional_version_details": {
              "dynamic": true,
              "type": "object"
            },
            "company_name": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "crypto_features": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "crypto_type": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "description": {
              "fields": {
                "keyword": {
                  "ignore_above": 512,
                  "type": "keyword"
                }
              },
              "type": "text"
            },
            "detected_apis": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "detection_time": {
              "type": "date"
            },
            "name": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "path": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "product_name": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "product_version": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "type": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "version": {
              "ignore_above": 256,
              "type": "keyword"
            }
          },
          "type": "object"
        },
        "network": {
          "properties": {
            "application": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "protocol": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "transport": {
              "ignore_above": 256,
              "type": "keyword"
            }
          },
          "type": "object"
        },
        "observer": {
          "properties": {
            "architecture": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "domain": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "hostname": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "id": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "ip": {
              "type": "ip"
            },
            "os": {
              "properties": {
                "family": {
                  "ignore_above": 256,
                  "type": "keyword"
                },
                "kernel": {
                  "ignore_above": 256,
                  "type": "keyword"
                },
                "name": {
                  "ignore_above": 256,
                  "type": "keyword"
                },
                "platform": {
                  "ignore_above": 256,
                  "type": "keyword"
                },
                "version": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              },
              "type": "object"
            },
            "type": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "vendor": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "version": {
              "ignore_above": 256,
              "type": "keyword"
            }
          },
          "type": "object"
        },
        "pe": {
          "properties": {
            "company": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "description": {
              "fields": {
                "keyword": {
                  "ignore_above": 512,
                  "type": "keyword"
                }
              },
              "type": "text"
            },
            "file_version": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "product": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "product_version": {
              "ignore_above": 256,
              "type": "keyword"
            }
          },
          "type": "object"
        },
        "pqc": {
          "properties": {
            "crypto_library": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "key_strength": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "last_assessed": {
              "type": "date"
            },
            "migration_priority": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "migration_status": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "quantum_resistance": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "quantum_risk": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "readiness": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "ready": {
              "type": "boolean"
            },
            "reason": {
              "fields": {
                "keyword": {
                  "ignore_above": 512,
                  "type": "keyword"
                }
              },
              "type": "text"
            },
            "recommended_action": {
              "fields": {
                "keyword": {
                  "ignore_above": 512,
                  "type": "keyword"
                }
              },
              "type": "text"
            },
            "supported_algorithms": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "vulnerable": {
              "type": "boolean"
            }
          },
          "type": "object"
        },
        "process": {
          "properties": {
            "command_line": {
              "fields": {
                "keyword": {
                  "ignore_above": 1024,
                  "type": "keyword"
                }
              },
              "type": "text"
            },
            "executable": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "hash": {
              "properties": {
                "md5": {
                  "ignore_above": 256,
                  "type": "keyword"
                },
                "sha1": {
                  "ignore_above": 256,
                  "type": "keyword"
                },
                "sha256": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              },
              "type": "object"
            },
            "name": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "owner": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "pid": {
              "type": "long"
            },
            "start": {
              "type": "date"
            },
            "user_name": {
              "ignore_above": 256,
              "type": "keyword"
            }
          },
          "type": "object"
        },
        "quantum_ready": {
          "type": "boolean"
        },
        "quantum_ready_cert": {
          "type": "boolean"
        },
        "quantum_ready_cipher": {
          "type": "boolean"
        },
        "quantum_ready_kx": {
          "type": "boolean"
        },
        "scan_mode": {
          "ignore_above": 256,
          "type": "keyword"
        },
        "scan_timestamp": {
          "type": "date"
        },
        "scan_type": {
          "ignore_above": 256,
          "type": "keyword"
        },
        "security": {
          "properties": {
            "known_vulnerabilities": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "last_assessed": {
              "type": "date"
            },
            "perfect_forward_secrecy": {
              "type": "boolean"
            },
            "pqc_support": {
              "type": "boolean"
            },
            "pqc_vulnerable": {
              "type": "boolean"
            },
            "risk_level": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "score": {
              "type": "long"
            },
            "vulnerable": {
              "type": "boolean"
            },
            "weak_crypto": {
              "type": "boolean"
            },
            "weak_dh_group": {
              "type": "boolean"
            }
          },
          "type": "object"
        },
        "security_association": {
          "properties": {
            "add_time": {
              "type": "long"
            },
            "auth_algorithm": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "current_bytes": {
              "type": "long"
            },
            "current_packets": {
              "type": "long"
            },
            "encryption_algorithm": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "hard_byte_limit": {
              "type": "long"
            },
            "hard_packet_limit": {
              "type": "long"
            },
            "lifetime_seconds": {
              "type": "long"
            },
            "mode": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "replay_window": {
              "type": "long"
            },
            "soft_byte_limit": {
              "type": "long"
            },
            "soft_packet_limit": {
              "type": "long"
            },
            "spi": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "state": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "use_time": {
              "type": "long"
            }
          },
          "type": "object"
        },
        "server": {
          "properties": {
            "address": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "domain": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "ip": {
              "type": "ip"
            },
            "port": {
              "type": "long"
            }
          },
          "type": "object"
        },
        "service": {
          "properties": {
            "name": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "type": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "version": {
              "ignore_above": 256,
              "type": "keyword"
            }
          },
          "type": "object"
        },
        "ssh": {
          "properties": {
            "banner": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "client_algorithms_offered": {
              "dynamic": true,
              "type": "object"
            },
            "handshake_error": {
              "fields": {
                "keyword": {
                  "ignore_above": 512,
                  "type": "keyword"
                }
              },
              "type": "text"
            },
            "server": {
              "properties": {
                "host_key": {
                  "properties": {
                    "bits": {
                      "type": "long"
                    },
                    "fingerprint_sha256": {
                      "ignore_above": 256,
                      "type": "keyword"
                    },
                    "type": {
                      "ignore_above": 256,
                      "type": "keyword"
                    }
                  },
                  "type": "object"
                }
              },
              "type": "object"
            }
          },
          "type": "object"
        },
        "tags": {
          "ignore_above": 256,
          "type": "keyword"
        },
        "tls": {
          "properties": {
            "certificate": {
              "properties": {
                "alias": {
                  "ignore_above": 256,
                  "type": "keyword"
                },
                "chain_complete": {
                  "type": "boolean"
                },
                "chain_length": {
                  "type": "long"
                },
                "extended_key_usage": {
                  "ignore_above": 256,
                  "type": "keyword"
                },
                "has_private_key": {
                  "type": "boolean"
                },
                "is_ca": {
                  "type": "boolean"
                },
                "is_self_signed": {
                  "type": "boolean"
                },
                "issuer": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "issuer_common_name": {
                  "ignore_above": 256,
                  "type": "keyword"
                },
                "key_usage": {
                  "ignore_above": 256,
                  "type": "keyword"
                },
                "not_after": {
                  "type": "date"
                },
                "not_before": {
                  "type": "date"
                },
                "serial_number": {
                  "ignore_above": 256,
                  "type": "keyword"
                },
                "subject": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "subject_common_name": {
                  "ignore_above": 256,
                  "type": "keyword"
                },
                "version": {
                  "type": "long"
                }
              },
              "type": "object"
            },
            "cipher": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "curve": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "private_key": {
              "properties": {
                "algorithm": {
                  "ignore_above": 256,
                  "type": "keyword"
                },
                "bit_size": {
                  "type": "long"
                },
                "curve": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              },
              "type": "object"
            },
            "server": {
              "properties": {
                "cipher": {
                  "ignore_above": 256,
                  "type": "keyword"
                },
                "host_key": {
                  "properties": {
                    "bits": {
                      "type": "long"
                    },
                    "fingerprint_sha256": {
                      "ignore_above": 256,
                      "type": "keyword"
                    },
                    "type": {
                      "ignore_above": 256,
                      "type": "keyword"
                    }
                  },
                  "type": "object"
                },
                "negotiated_group": {
                  "ignore_above": 256,
                  "type": "keyword"
                },
                "protocol_version": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              },
              "type": "object"
            },
            "version": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "version_protocol": {
              "ignore_above": 256,
              "type": "keyword"
            }
          },
          "type": "object"
        },
        "tunnel_details": {
          "properties": {
            "additional_params": {
              "dynamic": true,
              "type": "object"
            },
            "auth_method": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "bytes_received": {
              "type": "long"
            },
            "bytes_sent": {
              "type": "long"
            },
            "dh_group": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "encryption_algorithm": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "established_time": {
              "type": "date"
            },
            "expiry_time": {
              "type": "date"
            },
            "integrity_algorithm": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "local_endpoint": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "local_subnet": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "mode": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "nat_traversal": {
              "type": "boolean"
            },
            "perfect_forward_secrecy": {
              "type": "boolean"
            },
            "protocol": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "remote_endpoint": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "remote_subnet": {
              "ignore_above": 256,
              "type": "keyword"
            }
          },
          "type": "object"
        },
        "tychon": {
          "dynamic": true,
          "properties": {
            "active": {
              "type": "boolean"
            },
            "asset_type": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "key_strength": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "last_seen": {
              "type": "date"
            },
            "migration_priority": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "pqc_readiness": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "pqc_vulnerable": {
              "type": "boolean"
            },
            "quantum_risk": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "recommended_action": {
              "fields": {
                "keyword": {
                  "ignore_above": 512,
                  "type": "keyword"
                }
              },
              "type": "text"
            },
            "scan_mode": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "scan_timestamp": {
              "type": "date"
            },
            "scanner_version": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "type": {
              "ignore_above": 256,
              "type": "keyword"
            }
          },
          "type": "object"
        },
        "url": {
          "properties": {
            "domain": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "full": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "path": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "port": {
              "type": "long"
            },
            "scheme": {
              "ignore_above": 256,
              "type": "keyword"
            }
          },
          "type": "object"
        },
        "vpn_client": {
          "properties": {
            "active": {
              "type": "boolean"
            },
            "config_path": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "detection_confidence": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "detection_method": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "executable_path": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "first_detected": {
              "type": "date"
            },
            "install_path": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "last_seen": {
              "type": "date"
            },
            "name": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "process_id": {
              "type": "long"
            },
            "service_name": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "status": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "vendor": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "version": {
              "ignore_above": 256,
              "type": "keyword"
            }
          },
          "type": "object"
        },
        "vpn_config": {
          "properties": {
            "authentication_method": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "auto_reconnect": {
              "type": "boolean"
            },
            "config_encrypted": {
              "type": "boolean"
            },
            "dns_leak_protection": {
              "type": "boolean"
            },
            "ipv6_leak_protection": {
              "type": "boolean"
            },
            "kill_switch": {
              "type": "boolean"
            },
            "logging_enabled": {
              "type": "boolean"
            },
            "split_tunneling": {
              "type": "boolean"
            },
            "weak_settings": {
              "ignore_above": 256,
              "type": "keyword"
            }
          },
          "type": "object"
        },
        "vpn_connection": {
          "properties": {
            "authentication_algorithm": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "bytes_received": {
              "type": "long"
            },
            "bytes_sent": {
              "type": "long"
            },
            "cipher_suite": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "connected_since": {
              "type": "date"
            },
            "control_channel_auth": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "dh_group": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "dns_servers": {
              "type": "ip"
            },
            "encryption_algorithm": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "key_exchange_algorithm": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "local_ip": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "protocol": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "remote_ip": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "server_address": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "server_port": {
              "type": "long"
            },
            "tunnel_interface": {
              "ignore_above": 256,
              "type": "keyword"
            }
          },
          "type": "object"
        },
        "vulnerability": {
          "properties": {
            "category": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "cve_list": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "description": {
              "fields": {
                "keyword": {
                  "ignore_above": 512,
                  "type": "keyword"
                }
              },
              "type": "text"
            },
            "fixed_in_version": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "id": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "is_vulnerable": {
              "type": "boolean"
            },
            "name": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "risk_level": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "risk_reason": {
              "fields": {
                "keyword": {
                  "ignore_above": 512,
                  "type": "keyword"
                }
              },
              "type": "text"
            },
            "scanner": {
              "properties": {
                "vendor": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              },
              "type": "object"
            }
          },
          "type": "object"
        },
        "x509": {
          "properties": {
            "enhanced_key_usage": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "hash": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "is_self_signed": {
              "type": "boolean"
            },
            "is_valid": {
              "type": "boolean"
            },
            "issuer": {
              "properties": {
                "common_name": {
                  "ignore_above": 256,
                  "type": "keyword"
                },
                "country": {
                  "ignore_above": 256,
                  "type": "keyword"
                },
                "distinguished_name": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "locality": {
                  "ignore_above": 256,
                  "type": "keyword"
                },
                "organization": {
                  "ignore_above": 256,
                  "type": "keyword"
                },
                "organizational_unit": {
                  "ignore_above": 256,
                  "type": "keyword"
                },
                "state_or_province": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              },
              "type": "object"
            },
            "key_usage": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "not_after": {
              "type": "date"
            },
            "not_before": {
              "type": "date"
            },
            "public_key_algorithm": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "public_key_curve": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "public_key_size": {
              "type": "long"
            },
            "serial_number": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "signature_algorithm": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "subject": {
              "properties": {
                "common_name": {
                  "ignore_above": 256,
                  "type": "keyword"
                },
                "country": {
                  "ignore_above": 256,
                  "type": "keyword"
                },
                "distinguished_name": {
                  "ignore_above": 1024,
                  "type": "keyword"
                },
                "locality": {
                  "ignore_above": 256,
                  "type": "keyword"
                },
                "organization": {
                  "ignore_above": 256,
                  "type": "keyword"
                },
                "organizational_unit": {
                  "ignore_above": 256,
                  "type": "keyword"
                },
                "state_or_province": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              },
              "type": "object"
            },
            "subject_key_identifier": {
              "ignore_above": 256,
              "type": "keyword"
            },
            "validity": {
              "properties": {
                "not_after": {
                  "type": "date"
                },
                "not_before": {
                  "type": "date"
                }
              },
              "type": "object"
            },
            "version_number": {
              "type": "long"
            }
          },
          "type": "object"
        }
      }
    },
    "settings": {
      "index": {
        "codec": "best_compression",
        "mapping": {
          "total_fields": {
            "limit": 2000
          }
        },
        "number_of_replicas": 0,
        "number_of_shards": 1,
        "refresh_interval": "30s"
      }
    }
  }
}


# Verify deployment:
GET _index_template/tychon-pqc

Option B: Via Command Line

# Deploy the index template
curl -X PUT "https://your-elasticsearch:9200/_index_template/tychon-pqc" \
  -u "elastic:yourpassword" \
  -H "Content-Type: application/json" \
  -d @tychon-pqc-index-template.json

# Verify deployment
curl -X GET "https://your-elasticsearch:9200/_index_template/tychon-pqc" \
  -u "elastic:yourpassword"

✅ Expected Response

{
  "index_templates": [
    {
      "name": "tychon-pqc",
      "index_template": {
        "index_patterns": ["tychon-pqc-*", "logs-tychon-pqc-*"],
        "priority": 200,
        ...
      }
    }
  ]
}

Step 2: Deploy Ingest Pipeline

The ingest pipeline reconstructs nested objects from flat dot notation. This is critical for dashboard compatibility.

📥 Download the pipeline: ingest_pipeline.json

Via Kibana Dev Tools

Copy the entire block below and paste it into Kibana Dev Tools console, then click the green ▶ play button:

💡 Tip: Click inside the code block, press Ctrl+A (Cmd+A on Mac) to select all, then Ctrl+C (Cmd+C) to copy.

PUT _ingest/pipeline/tychon-pqc-flat-ndjson-pipeline
{
  "description": "Tychon PQC Flat NDJSON Ingest Pipeline - Reconstructs nested objects from flat dot notation fields",
  "processors": [
    {
      "set": {
        "description": "Set event.ingested timestamp",
        "field": "event.ingested",
        "value": "{{_ingest.timestamp}}"
      }
    },
    {
      "set": {
        "description": "Ensure @timestamp exists, copy from scan timestamp if missing",
        "field": "@timestamp",
        "value": "{{scan.timestamp}}",
        "if": "ctx['@timestamp'] == null && ctx?.scan?.timestamp != null"
      }
    },
    {
      "set": {
        "description": "Set @timestamp to ingest time if still missing",
        "field": "@timestamp",
        "value": "{{_ingest.timestamp}}",
        "if": "ctx['@timestamp'] == null"
      }
    },
    {
      "script": {
        "description": "Determine event type for routing based on fields present",
        "lang": "painless",
        "source": "if (ctx?.cipher?.cipher_suite != null) { ctx.event_type_routing = 'cipher'; } else if (ctx?.event?.action != null && ctx.event.action.contains('certificate')) { ctx.event_type_routing = 'certificate'; } else if (ctx?.event?.action != null && ctx.event.action.contains('keystore')) { ctx.event_type_routing = 'keystore'; } else if (ctx?.event?.action != null && ctx.event.action.contains('crypto_library')) { ctx.event_type_routing = 'process'; } else if (ctx?.event?.action != null && ctx.event.action.contains('vpn')) { ctx.event_type_routing = 'vpn'; } else if (ctx?.event?.action != null && ctx.event.action.contains('ipsec')) { ctx.event_type_routing = 'ipsec'; } else if (ctx?.quantum?.overall_score != null && ctx?.cipher?.cipher_suite == null) { ctx.event_type_routing = 'quantum'; }"
      }
    },
    {
      "script": {
        "description": "Reconstruct tychon.certificate_leaf_details nested object from certificate.leaf.* flat fields",
        "lang": "painless",
        "source": "if (ctx?.certificate?.leaf != null) { if (ctx.tychon == null) { ctx.tychon = [:]; } Map leafDetails = [:]; if (ctx.certificate.leaf.subject != null) { leafDetails.subject = ctx.certificate.leaf.subject; } if (ctx.certificate.leaf.issuer != null) { leafDetails.issuer = ctx.certificate.leaf.issuer; } if (ctx.certificate.leaf.validity != null) { leafDetails.validity = ctx.certificate.leaf.validity; } if (ctx.certificate.leaf.serial_number != null) { leafDetails.serial_number = ctx.certificate.leaf.serial_number; } if (ctx.certificate.leaf.signature_algorithm != null) { leafDetails.signature_algorithm = ctx.certificate.leaf.signature_algorithm; } if (ctx.certificate.leaf.version != null) { leafDetails.version = ctx.certificate.leaf.version; } if (ctx.certificate.leaf.is_self_signed != null) { leafDetails.is_self_signed = ctx.certificate.leaf.is_self_signed; } if (ctx.certificate.leaf.key_usage != null) { leafDetails.key_usage = ctx.certificate.leaf.key_usage; } if (ctx.certificate.leaf.extended_key_usage != null) { leafDetails.extended_key_usage = ctx.certificate.leaf.extended_key_usage; } if (ctx.certificate.leaf.basic_constraints != null) { leafDetails.basic_constraints = ctx.certificate.leaf.basic_constraints; } if (ctx.certificate.leaf.subject_public_key_info != null) { leafDetails.subject_public_key_info = ctx.certificate.leaf.subject_public_key_info; } if (ctx.certificate.leaf.sha256_fingerprint != null) { leafDetails.sha256_fingerprint = ctx.certificate.leaf.sha256_fingerprint; } if (ctx.certificate.leaf.sha1_fingerprint != null) { leafDetails.sha1_fingerprint = ctx.certificate.leaf.sha1_fingerprint; } if (ctx.certificate.leaf.pqc_vulnerable != null) { leafDetails.pqc_vulnerable = ctx.certificate.leaf.pqc_vulnerable; } if (ctx.certificate.leaf.raw_pem_certificate != null) { leafDetails.raw_pem_certificate = ctx.certificate.leaf.raw_pem_certificate; } if (ctx.certificate.leaf.source_id != null) { leafDetails.source_id = ctx.certificate.leaf.source_id; } ctx.tychon.certificate_leaf_details = leafDetails; }",
        "if": "ctx?.certificate?.leaf != null"
      }
    },
    {
      "script": {
        "description": "Reconstruct tychon.cipher_negotiation nested object from cipher.* flat fields",
        "lang": "painless",
        "source": "if (ctx?.cipher != null) { if (ctx.tychon == null) { ctx.tychon = [:]; } Map cipherNeg = [:]; if (ctx.cipher.cipher_suite != null) { cipherNeg.cipher_suite = ctx.cipher.cipher_suite; } if (ctx.cipher.protocol != null) { cipherNeg.protocol = ctx.cipher.protocol; } if (ctx.cipher.key_length_bits != null) { cipherNeg.key_length_bits = ctx.cipher.key_length_bits; } if (ctx.cipher.is_preferred != null) { cipherNeg.is_preferred = ctx.cipher.is_preferred; } if (ctx.cipher.active != null) { cipherNeg.active = ctx.cipher.active; } if (ctx.cipher.source != null) { cipherNeg.source = ctx.cipher.source; } if (ctx.cipher.source_id != null) { cipherNeg.source_id = ctx.cipher.source_id; } if (ctx.cipher.universal_id != null) { cipherNeg.universal_id = ctx.cipher.universal_id; } if (ctx.cipher.session_id != null) { cipherNeg.session_id = ctx.cipher.session_id; } if (ctx.cipher.compression_method != null) { cipherNeg.compression_method = ctx.cipher.compression_method; } if (ctx.cipher.alpn_protocol != null) { cipherNeg.alpn_protocol = ctx.cipher.alpn_protocol; } if (ctx.cipher.last_seen != null) { cipherNeg.last_seen = ctx.cipher.last_seen; } if (ctx.cipher.session_ticket_lifetime_hint_seconds != null) { cipherNeg.session_ticket_lifetime_hint_seconds = ctx.cipher.session_ticket_lifetime_hint_seconds; } if (ctx.cipher.intel != null) { cipherNeg.intel = ctx.cipher.intel; } ctx.tychon.cipher_negotiation = cipherNeg; }",
        "if": "ctx?.cipher != null"
      }
    },
    {
      "script": {
        "description": "Convert cipher vulnerabilities from comma-separated string to array",
        "lang": "painless",
        "source": "if (ctx?.tychon?.cipher_negotiation?.intel?.vulnerabilities != null && ctx.tychon.cipher_negotiation.intel.vulnerabilities instanceof String) { String vulnString = ctx.tychon.cipher_negotiation.intel.vulnerabilities; List vulnList = new ArrayList(); String[] parts = vulnString.splitOnToken(','); for (String part : parts) { String trimmed = part.trim(); if (trimmed.length() > 0) { vulnList.add(trimmed); } } ctx.tychon.cipher_negotiation.intel.vulnerabilities = vulnList; }",
        "if": "ctx?.tychon?.cipher_negotiation?.intel?.vulnerabilities != null"
      }
    },
    {
      "set": {
        "description": "Map certificate.leaf fields to x509 namespace for ECS compliance",
        "field": "x509.subject.common_name",
        "value": "{{certificate.leaf.subject.common_name}}",
        "if": "ctx?.certificate?.leaf?.subject?.common_name != null"
      }
    },
    {
      "set": {
        "description": "Map certificate issuer common name to x509",
        "field": "x509.issuer.common_name",
        "value": "{{certificate.leaf.issuer.common_name}}",
        "if": "ctx?.certificate?.leaf?.issuer?.common_name != null"
      }
    },
    {
      "set": {
        "description": "Map certificate issuer organization to x509",
        "field": "x509.issuer.organization",
        "value": "{{certificate.leaf.issuer.organization}}",
        "if": "ctx?.certificate?.leaf?.issuer?.organization != null"
      }
    },
    {
      "set": {
        "description": "Map certificate subject organization to x509",
        "field": "x509.subject.organization",
        "value": "{{certificate.leaf.subject.organization}}",
        "if": "ctx?.certificate?.leaf?.subject?.organization != null"
      }
    },
    {
      "set": {
        "description": "Map certificate serial number to x509",
        "field": "x509.serial_number",
        "value": "{{certificate.leaf.serial_number}}",
        "if": "ctx?.certificate?.leaf?.serial_number != null"
      }
    },
    {
      "set": {
        "description": "Map certificate signature algorithm to x509",
        "field": "x509.signature_algorithm",
        "value": "{{certificate.leaf.signature_algorithm}}",
        "if": "ctx?.certificate?.leaf?.signature_algorithm != null"
      }
    },
    {
      "set": {
        "description": "Map certificate validity not_before to x509",
        "field": "x509.not_before",
        "value": "{{certificate.leaf.validity.not_before}}",
        "if": "ctx?.certificate?.leaf?.validity?.not_before != null"
      }
    },
    {
      "set": {
        "description": "Map certificate validity not_after to x509",
        "field": "x509.not_after",
        "value": "{{certificate.leaf.validity.not_after}}",
        "if": "ctx?.certificate?.leaf?.validity?.not_after != null"
      }
    },
    {
      "set": {
        "description": "Map certificate public key algorithm to x509",
        "field": "x509.public_key_algorithm",
        "value": "{{certificate.leaf.subject_public_key_info.algorithm}}",
        "if": "ctx?.certificate?.leaf?.subject_public_key_info?.algorithm != null"
      }
    },
    {
      "set": {
        "description": "Map certificate public key size to x509",
        "field": "x509.public_key_size",
        "value": "{{certificate.leaf.subject_public_key_info.bit_size}}",
        "if": "ctx?.certificate?.leaf?.subject_public_key_info?.bit_size != null"
      }
    },
    {
      "set": {
        "description": "Map certificate public key curve to x509",
        "field": "x509.public_key_curve",
        "value": "{{certificate.leaf.subject_public_key_info.curve}}",
        "if": "ctx?.certificate?.leaf?.subject_public_key_info?.curve != null"
      }
    },
    {
      "date": {
        "description": "Parse certificate not_before date",
        "field": "certificate.leaf.validity.not_before",
        "target_field": "certificate.leaf.validity.not_before",
        "formats": [
          "ISO8601",
          "yyyy-MM-dd'T'HH:mm:ssXXX",
          "yyyy-MM-dd'T'HH:mm:ss.SSSXXX",
          "yyyy-MM-dd HH:mm:ss",
          "MMM dd HH:mm:ss yyyy 'GMT'"
        ],
        "if": "ctx?.certificate?.leaf?.validity?.not_before != null",
        "on_failure": [
          {
            "append": {
              "field": "error.pipeline",
              "value": "Failed to parse certificate.leaf.validity.not_before: {{_ingest.on_failure_message}}"
            }
          }
        ]
      }
    },
    {
      "date": {
        "description": "Parse certificate not_after date",
        "field": "certificate.leaf.validity.not_after",
        "target_field": "certificate.leaf.validity.not_after",
        "formats": [
          "ISO8601",
          "yyyy-MM-dd'T'HH:mm:ssXXX",
          "yyyy-MM-dd'T'HH:mm:ss.SSSXXX",
          "yyyy-MM-dd HH:mm:ss",
          "MMM dd HH:mm:ss yyyy 'GMT'"
        ],
        "if": "ctx?.certificate?.leaf?.validity?.not_after != null",
        "on_failure": [
          {
            "append": {
              "field": "error.pipeline",
              "value": "Failed to parse certificate.leaf.validity.not_after: {{_ingest.on_failure_message}}"
            }
          }
        ]
      }
    },
    {
      "date": {
        "description": "Parse cipher last_seen date",
        "field": "cipher.last_seen",
        "target_field": "cipher.last_seen",
        "formats": [
          "ISO8601",
          "yyyy-MM-dd'T'HH:mm:ssXXX",
          "yyyy-MM-dd'T'HH:mm:ss.SSSXXX"
        ],
        "if": "ctx?.cipher?.last_seen != null",
        "on_failure": [
          {
            "append": {
              "field": "error.pipeline",
              "value": "Failed to parse cipher.last_seen: {{_ingest.on_failure_message}}"
            }
          }
        ]
      }
    },
    {
      "set": {
        "description": "Set event.dataset to default if missing",
        "field": "event.dataset",
        "value": "tychon.pqc",
        "if": "ctx?.event?.dataset == null"
      }
    },
    {
      "set": {
        "description": "Set event.kind to event if missing",
        "field": "event.kind",
        "value": "event",
        "if": "ctx?.event?.kind == null"
      }
    }
  ],
  "on_failure": [
    {
      "set": {
        "description": "Set error flag on pipeline failure",
        "field": "error.pipeline",
        "value": "Pipeline processing failed: {{_ingest.on_failure_message}}"
      }
    },
    {
      "set": {
        "description": "Preserve the failed processor",
        "field": "error.processor",
        "value": "{{_ingest.on_failure_processor_type}}"
      }
    }
  ]
}

# Verify deployment:
GET _ingest/pipeline/tychon-pqc-flat-ndjson-pipeline

Via Command Line

curl -X PUT "https://your-elasticsearch:9200/_ingest/pipeline/tychon-pqc-flat-ndjson-pipeline" \
  -u "elastic:yourpassword" \
  -H "Content-Type: application/json" \
  -d @ingest_pipeline.json

Test Pipeline (Recommended):

Test with sample data before deploying to production:

ℹ️ Note: The test below uses nested objects (not flat dot notation) because the NDJSON parser expands dots before the pipeline runs. Your flat NDJSON file has "cipher.intel.vulnerabilities": "..." but Filestream converts it to cipher: { intel: { vulnerabilities: "..." } } before processing.

POST _ingest/pipeline/tychon-pqc-flat-ndjson-pipeline/_simulate
{
  "docs": [
    {
      "_source": {
        "certificate": {
          "leaf": {
            "subject": {
              "common_name": "test.example.com"
            },
            "issuer": {
              "organization": "Test CA"
            }
          }
        },
        "cipher": {
          "cipher_suite": "TLS_AES_256_GCM_SHA384",
          "protocol": "TLSv1.3",
          "intel": {
            "vulnerabilities": "ROBOT, BEAST"
          }
        }
      }
    }
  ]
}

✅ Verify Output Includes:

• tychon.certificate_leaf_details.subject.common_name: "test.example.com"
• tychon.cipher_negotiation.cipher_suite: "TLS_AES_256_GCM_SHA384"
• tychon.cipher_negotiation.intel.vulnerabilities: ["ROBOT", "BEAST"] (array, not string)
• event_type_routing: "cipher"

Step 3: Configure Custom Logs (Filestream) Integration

This integration monitors NDJSON files on the filesystem and sends them to Elasticsearch.

Kibana UI Configuration:

Navigate to Fleet Integrations Page:
- Open Kibana → Management → Integrations
- Click Integrations in left sidebar
Add Custom Logs Integration:
- Search for "Custom Logs"
- Click Custom Logs (Filestream) integration card
- Click Add Custom Logs (Filestream) button on the top right of the screen

Configure Settings:

Integration name:	`tychon-pqc-flat-ndjson`
Dataset name:	`tychon-pqc` ⚠️ Must be exact
Log file path:	`/path/to/scanner/output/*.ndjson`

⚠️ CRITICAL: Custom Ingest Pipeline (set this BEFORE configuring parsers):

Under "Custom configurations" section, set:

Custom ingest pipeline: tychon-pqc-flat-ndjson-pipeline
Parsers Configuration:
Parsers (text box - add this YAML configuration):
```
- ndjson:
    overwrite_keys: true
    add_error_key: true
```

Advanced Options (optional but recommended):

Clean removed:	✅ Enabled (removes processed files)
Ignore older than:	`24h` (optional - skip old files)

Choose Agent Policy:
- Select policy deployed on scanner host(s)
- Click Save and continue

📋 Example File Paths

Linux: /var/log/tychon-pqc/*.ndjson
macOS: /usr/local/var/log/tychon-pqc/*.ndjson
Windows: C:\ProgramData\Tychon\output\*.ndjson

Step 4: Deploy Transforms

Transforms route data from the data stream to specialized indices with upsert capability.

📥 Download all transform files:

Deploy All 7 Transforms:

# Deploy cipher transform
curl -X PUT "https://your-elasticsearch:9200/_transform/tychon-pqc-cipher-transform" \
  -u "elastic:yourpassword" \
  -H "Content-Type: application/json" \
  -d @transform-cipher.json

# Deploy certificate transform
curl -X PUT "https://your-elasticsearch:9200/_transform/tychon-pqc-certificate-transform" \
  -u "elastic:yourpassword" \
  -H "Content-Type: application/json" \
  -d @transform-certificate.json

# Repeat for all 7 transforms...
# (keystore, process, vpn, ipsec, quantum)

⚠️ Start Transforms (After Data Flows):

Wait until data is flowing into the data stream before starting transforms.

# Start all transforms
curl -X POST "https://your-elasticsearch:9200/_transform/tychon-pqc-cipher-transform/_start" \
  -u "elastic:yourpassword"

curl -X POST "https://your-elasticsearch:9200/_transform/tychon-pqc-certificate-transform/_start" \
  -u "elastic:yourpassword"

# Repeat for all 7 transforms...

Verify Transforms Running:

GET _transform/_stats

✅ Expected: Each transform shows

• state: "started"
• health.status: "green"

Step 5: Configure Scanner Output

Configure TYCHON to write flat NDJSON files to the directory monitored by Filestream.

Scanner Command Examples:

# Linux/macOS
./certscanner-linux-amd64 \
  -mode local \
  -scanfilesystem -scanconnected \
  -output /var/log/tychon-pqc/scan-$(date +%Y%m%d-%H%M%S).ndjson \
  -outputformat flat-ndjson

# Windows
.\certscanner-windows-amd64.exe `
  -mode local `
  -scanfilesystem -scanconnected `
  -output "C:\ProgramData\Tychon\output\scan-$((Get-Date).ToString('yyyyMMdd-HHmmss')).ndjson" `
  -outputformat flat-ndjson

💡 Best Practices

• Use timestamped filenames to avoid overwriting
• Ensure directory exists and has write permissions
• Ensure Elastic Agent has read permissions on files
• Use .ndjson file extension to match glob pattern

Step 6: Verify Deployment

1. Check Data Stream:

GET logs-tychon-pqc-default/_count

2. Check Nested Object Reconstruction:

GET tychon-pqc-ciphers/_search
{
  "size": 1,
  "query": { "match_all": {} }
}

Verify response includes tychon.cipher_negotiation nested object.

3. Check Transform Routing:

# Check document counts in each index
GET _cat/indices/tychon-pqc-*?v

4. Verify Dashboard:

Open Kibana → Dashboard → "Tychon Quantum Readiness"
Verify all visualizations load without errors
Check that data appears in charts

Troubleshooting

No Data in Data Stream

Check:

Files exist in monitored directory: ls /path/to/scanner/output/*.ndjson
Elastic Agent has read permissions on files
Check agent logs: /var/log/elastic-agent/elastic-agent.log
Verify file path in integration matches actual path

No Data in Destination Indices

Check:

Transforms are started: GET _transform/_stats
Transforms show state: "started"
Check transform health: health.status: "green"
Data exists in data stream: GET logs-tychon-pqc-default/_count

Mapping Errors in Dashboard

Check:

Nested objects exist: GET tychon-pqc-ciphers/_search → check for tychon.cipher_negotiation
Pipeline is processing correctly: GET _ingest/pipeline/tychon-pqc-flat-ndjson-pipeline/_stats
Check for pipeline errors: GET logs-tychon-pqc-*/_search?q=error.pipeline:*
Verify index template applied: GET tychon-pqc-ciphers/_mapping

High Data Stream Storage

Check:

Transforms are running: GET _transform/_stats
Transforms should delete from data stream after 7 days
Manually delete old data if needed: POST logs-tychon-pqc-*/_delete_by_query with date range

Data Type Separation Indexing

🚀 Enterprise-Scale Index Strategy

Both ingestion methods automatically separate data into specialized indexes based on asset type. This strategy enables optimized storage, queries, and management for large-scale deployments.

Index Types Created

tychon-pqc-ciphers-{suffix}

tychon-pqc-certificates-{suffix}

tychon-pqc-keystores-{suffix}

tychon-pqc-ssh-keys-{suffix}

tychon-pqc-crypto-libraries-{suffix}

tychon-pqc-processes-{suffix}

tychon-pqc-vpn-{suffix}

tychon-pqc-ipsec-{suffix}

Key Benefits

✅ Optimized Storage: Each index tailored for specific data types
✅ Faster Queries: Targeted searches without cross-type filtering
✅ Scalable Architecture: No date-based index creation
✅ Upsert Support: Consistent index names enable document updates
✅ Lifecycle Management: Independent retention policies per data type
✅ Custom Suffixes: Environment/deployment-specific organization

How Routing Works

Direct Posting Method:

• Scanner analyzes document content
• Routes to appropriate index via Bulk API
• Automatic based on field presence

Flat NDJSON Method:

• Ingest pipeline sets event_type_routing
• Transforms filter on routing field
• Automatic based on field presence + event.action

Index Template and Mapping

Both ingestion methods use the same index template with ECS-compliant field mappings. The template is automatically created by direct posting, but must be manually deployed for flat NDJSON integration.

📥 Download the complete template: tychon-pqc-index-template.json

Template Patterns

• Index Patterns: tychon-pqc-*, logs-tychon-pqc-*
• Priority: 200 (high priority)
• Shards: 1 (adjust for production)
• Replicas: 0 (adjust for production)
• Codec: best_compression
• Refresh Interval: 30s
• Field Limit: 2000

Critical Mappings

Field	Type	Purpose	Full	Standard	Minimal
_id	keyword	Document upserts	✓	✓	✗
@timestamp	date	Event timestamp	✓	✓	✗
tychon.certificate_leaf_details	nested	Certificate details (dashboard critical)	✓	✓	✗
tychon.cipher_negotiation	nested	Cipher details (dashboard critical)	✓	✓	✗
observer.hostname	keyword	Scanner host	✓	✓	✗
event.action	keyword	Event type routing	✓	✓	✗

ECS Compliance

The template follows Elastic Common Schema (ECS) standards:

• observer.* - Scanner/observer information
• event.* - Event metadata (kind, category, action)
• host.* - Host information
• server.* - Server connection details
• x509.* - Certificate fields (ECS namespace)
• tls.* - TLS connection details
• ssh.* - SSH connection details
• tychon.* - Custom namespace for TYCHON-specific fields

Data Schema

Nested Object Structures

The dashboard requires specific nested object structures. Both ingestion methods produce identical nested objects:

tychon.certificate_leaf_details (nested)

{
  "subject": {
    "common_name": "example.com",
    "organization": "Example Corp",
    "country": "US"
  },
  "issuer": {
    "common_name": "Example CA",
    "organization": "Example CA Inc"
  },
  "subject_public_key_info": {
    "algorithm": "RSA",
    "bit_size": 2048
  },
  "validity": {
    "not_before": "2024-01-01T00:00:00Z",
    "not_after": "2025-01-01T00:00:00Z"
  }
}

tychon.cipher_negotiation (nested)

{
  "cipher_suite": "TLS_AES_256_GCM_SHA384",
  "protocol": "TLSv1.3",
  "key_length_bits": 256,
  "intel": {
    "vulnerabilities": ["ROBOT", "BEAST"],  // Array of strings
    "is_quantum_ready": false,
    "quantum_risk_level": "high"
  }
}

⚠️ Important: Nested Objects Required

The dashboard REQUIRES these fields to be nested objects, not flat dot notation. This is why flat NDJSON integration uses an ingest pipeline to reconstruct nested structures.

✅ Correct: tychon.certificate_leaf_details.subject.common_name (nested object)
❌ Wrong: tychon.certificate_leaf_details.subject.common_name (flat field with dots)

Query Examples

These queries work with data from both ingestion methods since both write to tychon-pqc-* indices. Replace {suffix} with your actual suffix:

1. Cross-Index Security Analysis

Find PQC-Vulnerable Assets Across All Data Types

GET tychon-pqc-*-{suffix}/_search
{
  "query": {
    "bool": {
      "must": [
        { "term": { "pqc.vulnerable": true } }
      ]
    }
  },
  "aggs": {
    "by_index": {
      "terms": {
        "field": "_index",
        "size": 20
      },
      "aggs": {
        "by_host": {
          "terms": {
            "field": "observer.hostname",
            "size": 10
          }
        }
      }
    },
    "asset_types": {
      "terms": {
        "field": "asset_type",
        "size": 10
      }
    }
  },
  "sort": [
    { "@timestamp": { "order": "desc" } }
  ]
}

Certificate Expiration Analysis

GET tychon-pqc-certificates-{suffix}/_search
{
  "query": {
    "bool": {
      "must": [
        { "exists": { "field": "tls.certificate.not_after" } },
        {
          "range": {
            "tls.certificate.not_after": {
              "gte": "now",
              "lte": "now+90d"
            }
          }
        }
      ]
    }
  },
  "aggs": {
    "expiring_soon": {
      "date_histogram": {
        "field": "tls.certificate.not_after",
        "calendar_interval": "week",
        "format": "yyyy-MM-dd"
      },
      "aggs": {
        "hosts": {
          "terms": {
            "field": "observer.hostname",
            "size": 10
          }
        }
      }
    }
  },
  "sort": [
    { "tls.certificate.not_after": { "order": "asc" } }
  ]
}

Weak Cipher Detection

GET tychon-pqc-ciphers-{suffix}/_search
{
  "query": {
    "bool": {
      "should": [
        { "wildcard": { "tls.cipher": "*RC4*" } },
        { "wildcard": { "tls.cipher": "*DES*" } },
        { "wildcard": { "tls.server.cipher": "*MD5*" } },
        { "wildcard": { "tls.server.cipher": "*SHA1*" } },
        { "term": { "quantum_ready_cipher": false } }
      ],
      "minimum_should_match": 1
    }
  },
  "aggs": {
    "weak_ciphers": {
      "terms": {
        "field": "tls.server.cipher",
        "size": 20
      }
    },
    "protocol_versions": {
      "terms": {
        "field": "tls.server.protocol_version",
        "size": 10
      }
    },
    "affected_hosts": {
      "terms": {
        "field": "observer.hostname",
        "size": 50
      }
    }
  }
}

2. Keystore-Specific Analysis

Keystore Vulnerability Summary

GET tychon-pqc-keystores-{suffix}/_search
{
  "query": {
    "bool": {
      "must": [
        { "term": { "event.action": "keystore_discovered" } }
      ]
    }
  },
  "aggs": {
    "keystore_types": {
      "terms": {
        "field": "keystore.type",
        "size": 10
      },
      "aggs": {
        "total_vulnerable_certs": {
          "sum": {
            "field": "keystore.stats.vulnerable_certificates"
          }
        },
        "total_pqc_vulnerable_certs": {
          "sum": {
            "field": "keystore.stats.pqc_vulnerable_certificates"
          }
        },
        "avg_cert_count": {
          "avg": {
            "field": "keystore.cert_count"
          }
        }
      }
    }
  }
}

3. SSH Key Analysis

SSH Host Key Security Assessment

GET tychon-pqc-ssh-keys-{suffix}/_search
{
  "query": {
    "bool": {
      "must": [
        { "exists": { "field": "ssh.server.host_key" } }
      ]
    }
  },
  "aggs": {
    "key_types": {
      "terms": {
        "field": "ssh.server.host_key.type",
        "size": 20
      },
      "aggs": {
        "avg_key_size": {
          "avg": {
            "field": "ssh.server.host_key.bits"
          }
        },
        "quantum_ready_count": {
          "filter": {
            "term": { "quantum_ready": true }
          }
        }
      }
    },
    "key_size_distribution": {
      "histogram": {
        "field": "ssh.server.host_key.bits",
        "interval": 512
      }
    }
  }
}

4. Process and Library Analysis

Cryptographic Library Discovery

GET tychon-pqc-processes-{suffix}/_search
{
  "query": {
    "bool": {
      "must": [
        { "term": { "event.action": "crypto_library_in_memory" } }
      ]
    }
  },
  "aggs": {
    "library_types": {
      "terms": {
        "field": "library.name",
        "size": 20
      },
      "aggs": {
        "versions": {
          "terms": {
            "field": "library.version",
            "size": 10
          }
        }
      }
    },
    "vulnerable_libraries": {
      "filter": {
        "term": { "vulnerability.is_vulnerable": true }
      },
      "aggs": {
        "by_library": {
          "terms": {
            "field": "library.name",
            "size": 10
          }
        }
      }
    }
  }
}

Dashboard

Tychon Quantum Readiness Dashboard

Both ingestion methods work with the same dashboard. No modifications are needed regardless of which method you use.

Import Dashboard

Via Kibana UI:

Navigate to Management → Stack Management → Saved Objects
Click Import
Select file: elastic_integration/certscanner/tychon_quantum_readiness_elastic_direct_2.0.ndjson
Click Import
If conflicts exist, choose "Overwrite"

Via Elasticsearch API:

curl -X POST "https://your-kibana:5601/api/saved_objects/_import" \
  -u "elastic:yourpassword" \
  -H "kbn-xsrf: true" \
  --form file=@tychon_quantum_readiness_elastic_direct_2.0.ndjson

Dashboard Features

Visualizations Include:

• Quantum vulnerability overview
• Certificate expiration timeline
• Cipher suite distribution
• SSH key algorithm breakdown
• Keystore vulnerability statistics
• Process crypto library discovery

Data Sources:

• tychon-pqc-ciphers-*
• tychon-pqc-certificates-*
• tychon-pqc-keystores-*
• tychon-pqc-ssh-keys-*
• tychon-pqc-processes-*
• tychon-pqc-vpn-*
• tychon-pqc-ipsec-*

Data Volume & Performance

Capacity Planning

Per-Endpoint Storage Estimate:

Remote TLS scan: ~1-5 KB/endpoint
Local filesystem scan: ~50-200 KB/endpoint
Comprehensive scan: ~100-500 KB/endpoint

Example Deployment (10,000 endpoints):

Initial scan: ~500 MB - 2 GB
Daily rescans: ~100 MB - 500 MB
Monthly growth: ~3-15 GB

Performance Tuning

For High Volume (>10,000 docs/sec):

Increase index shards: 3-5
Reduce refresh interval: 10s
Increase bulk batch size

For Low Volume (<100 docs/sec):

Use 1 shard per index
Increase refresh interval: 60s
Enable best compression

Lifecycle Management Recommendations

Index Management

• Use Index Lifecycle Management (ILM)
• Configure hot-warm-cold architecture
• Set replica shards based on query load
• Enable best_compression codec

Query Optimization

• Target specific index types vs wildcards
• Use bool queries with filters
• Implement date range filtering
• Leverage aggregations for statistics

Security

• Configure field-level security
• Use index patterns for RBAC
• Enable audit logging
• Implement network encryption

Monitoring

• Set Watcher alerts for cert expiration
• Monitor quantum vulnerability trends
• Create dashboards per data type
• Implement SLA monitoring