← Back to Documentation
AWS

AWS S3 Integration

Centralized cryptographic asset report storage and management

S3 Integration Overview

AWS S3 Centralized Storage

TYCHON Quantum Readiness can upload scan reports directly to AWS S3 for centralized storage, enabling automated processing, compliance archiving, and integration with AWS analytics services.

☁️ Cloud Storage

Automatic upload to S3 with organized folder structure

🔄 Automated Processing

Trigger Lambda functions and analytics pipelines

📋 Compliance Archive

Long-term retention for audit and compliance requirements

AWS Configuration

1. IAM Policy Configuration

Minimal IAM Policy for TYCHON Quantum Readiness

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:PutObjectAcl",
        "s3:GetBucketLocation"
      ],
      "Resource": [
        "arn:aws:s3:::your-certscanner-bucket",
        "arn:aws:s3:::your-certscanner-bucket/*"
      ]
    },
    {
      "Effect": "Allow", 
      "Action": [
        "s3:HeadBucket"
      ],
      "Resource": "arn:aws:s3:::your-certscanner-bucket"
    }
  ]
}

Create IAM User or Role

# Create IAM user for TYCHON Quantum Readiness
aws iam create-user --user-name certscanner-s3-uploader

# Attach the policy
aws iam attach-user-policy --user-name certscanner-s3-uploader \
  --policy-arn arn:aws:iam::YOUR_ACCOUNT:policy/TYCHON Quantum ReadinessS3Policy

# Create access keys
aws iam create-access-key --user-name certscanner-s3-uploader

2. S3 Bucket Configuration

Create S3 Bucket

# Create bucket with versioning and encryption
aws s3api create-bucket --bucket your-certscanner-reports \
  --region us-east-1

# Enable versioning
aws s3api put-bucket-versioning --bucket your-certscanner-reports \
  --versioning-configuration Status=Enabled

# Enable server-side encryption
aws s3api put-bucket-encryption --bucket your-certscanner-reports \
  --server-side-encryption-configuration '{
    "Rules": [
      {
        "ApplyServerSideEncryptionByDefault": {
          "SSEAlgorithm": "AES256"
        }
      }
    ]
  }'

Lifecycle Policy (Optional)

{
  "Rules": [
    {
      "ID": "TYCHON Quantum ReadinessReportsLifecycle",
      "Status": "Enabled",
      "Filter": {
        "Prefix": "certscanner/"
      },
      "Transitions": [
        {
          "Days": 30,
          "StorageClass": "STANDARD_IA"
        },
        {
          "Days": 90,
          "StorageClass": "GLACIER"
        },
        {
          "Days": 2555,
          "StorageClass": "DEEP_ARCHIVE"
        }
      ],
      "Expiration": {
        "Days": 2555
      }
    }
  ]
}

3. AWS Credentials Configuration

Multiple Authentication Methods

TYCHON Quantum Readiness supports multiple AWS authentication methods in order of precedence:

  1. 1. Command line flags: -s3accesskey and -s3secretkey
  2. 2. Environment variables: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
  3. 3. AWS credentials file: ~/.aws/credentials
  4. 4. IAM roles: EC2 instance profiles or ECS task roles

Command Line Flags

# Direct credential specification
./certscanner -host example.com \
  -upload-s3 -s3bucket "secure-bucket" \
  -s3accesskey "AKIAIOSFODNN7EXAMPLE" \
  -s3secretkey "wJalrXUtnFEMI/K7MDENG..."

# Useful for automation scripts
# with credential management

Environment Variables

# Set AWS credentials
export AWS_ACCESS_KEY_ID="your-access-key"
export AWS_SECRET_ACCESS_KEY="your-secret-key"
export AWS_DEFAULT_REGION="us-east-1"

# Optional: Use AWS profile
export AWS_PROFILE="certscanner"

IAM Roles (Recommended)

# No explicit credentials needed
# Uses EC2 instance profile or ECS task role
./certscanner -host example.com \
  -upload-s3 -s3bucket "secure-bucket"

# Automatic credential detection
# Most secure for production deployments

AWS Credentials File

# ~/.aws/credentials
[default]
aws_access_key_id = your-access-key
aws_secret_access_key = your-secret-key

[certscanner]
aws_access_key_id = your-access-key  
aws_secret_access_key = your-secret-key

# ~/.aws/config
[default]
region = us-east-1

[profile certscanner]
region = us-east-1

S3-Compatible Services

Cloudflare R2 Integration

TYCHON Quantum Readiness supports Cloudflare R2 and other S3-compatible storage services using custom endpoint URLs.

🌐 Cloudflare R2

Global edge storage with zero egress fees

-s3endpoint "https://account-id.r2.cloudflarestorage.com"

🗄️ MinIO

Self-hosted S3-compatible object storage

-s3endpoint "https://minio.company.com:9000"

Usage Examples

Basic S3 Upload

# Simple scan with S3 upload
.\certscanner-windows-amd64.exe -host example.com -upload-s3 -s3bucket "my-security-reports"

# Scan with custom region and prefix
.\certscanner-windows-amd64.exe -host 192.168.1.0/24 -cipherscan `
  -upload-s3 -s3bucket "company-certscanner" `
  -s3region "us-west-2" -s3prefix "production/network-scans"

# Local system scan to S3
.\certscanner-windows-amd64.exe -mode local -scanfilesystem -scanconnected `
  -outputformat flatndjson `
  -upload-s3 -s3bucket "security-reports" -s3prefix "endpoints"

# Upload with explicit AWS credentials for secured buckets
.\certscanner-windows-amd64.exe -host example.com -cipherscan `
  -upload-s3 -s3bucket "secure-certscanner-bucket" `
  -s3accesskey "AKIAIOSFODNN7EXAMPLE" `
  -s3secretkey "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" `
  -s3region "us-west-2" -s3prefix "production"

# Upload to Cloudflare R2 (S3-compatible)
.\certscanner-windows-amd64.exe -host example.com -cipherscan `
  -upload-s3 -s3bucket "cert-scanner" `
  -s3accesskey "your-r2-access-key" `
  -s3secretkey "your-r2-secret-key" `
  -s3endpoint "https://account-id.r2.cloudflarestorage.com" `
  -s3region "auto" -s3prefix "production"
# Simple scan with S3 upload
./certscanner-linux-x64 -host example.com -upload-s3 -s3bucket "my-security-reports"

# Scan with custom region and prefix
./certscanner-linux-x64 -host 192.168.1.0/24 -cipherscan \
  -upload-s3 -s3bucket "company-certscanner" \
  -s3region "us-west-2" -s3prefix "production/network-scans"

# Local system scan to S3
./certscanner-linux-x64 -mode local -scanfilesystem -scanconnected \
  -outputformat flatndjson \
  -upload-s3 -s3bucket "security-reports" -s3prefix "endpoints"

# Upload with explicit AWS credentials for secured buckets
./certscanner-linux-x64 -host example.com -cipherscan \
  -upload-s3 -s3bucket "secure-certscanner-bucket" \
  -s3accesskey "AKIAIOSFODNN7EXAMPLE" \
  -s3secretkey "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" \
  -s3region "us-west-2" -s3prefix "production"

# Upload to Cloudflare R2 (S3-compatible)
./certscanner-linux-x64 -host example.com -cipherscan \
  -upload-s3 -s3bucket "cert-scanner" \
  -s3accesskey "your-r2-access-key" \
  -s3secretkey "your-r2-secret-key" \
  -s3endpoint "https://account-id.r2.cloudflarestorage.com" \
  -s3region "auto" -s3prefix "production"
# Simple scan with S3 upload
./certscanner-darwin-amd64 -host example.com -upload-s3 -s3bucket "my-security-reports"
# Apple Silicon: ./certscanner-darwin-arm64 -host example.com -upload-s3 -s3bucket "my-security-reports"

# Scan with custom region and prefix
./certscanner-darwin-amd64 -host 192.168.1.0/24 -cipherscan \
  -upload-s3 -s3bucket "company-certscanner" \
  -s3region "us-west-2" -s3prefix "production/network-scans"

# Local system scan to S3 (no memory scanning on macOS)
./certscanner-darwin-amd64 -mode local -scanfilesystem -scanconnected \
  -outputformat flatndjson \
  -upload-s3 -s3bucket "security-reports" -s3prefix "endpoints"

# Upload with explicit AWS credentials for secured buckets
./certscanner-darwin-amd64 -host example.com -cipherscan \
  -upload-s3 -s3bucket "secure-certscanner-bucket" \
  -s3accesskey "AKIAIOSFODNN7EXAMPLE" \
  -s3secretkey "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" \
  -s3region "us-west-2" -s3prefix "production"

# Upload to Cloudflare R2 (S3-compatible)
./certscanner-darwin-amd64 -host example.com -cipherscan \
  -upload-s3 -s3bucket "cert-scanner" \
  -s3accesskey "your-r2-access-key" \
  -s3secretkey "your-r2-secret-key" \
  -s3endpoint "https://account-id.r2.cloudflarestorage.com" \
  -s3region "auto" -s3prefix "production"

Organized S3 Structure

# Production environment scans
.\certscanner-windows-amd64.exe -host prod-servers.txt -cipherscan `
  -tags "production,compliance" `
  -upload-s3 -s3bucket "certscanner-reports" `
  -s3prefix "environments/production"

# Development environment scans  
.\certscanner-windows-amd64.exe -host dev-servers.txt -quickscan `
  -tags "development,testing" `
  -upload-s3 -s3bucket "certscanner-reports" `
  -s3prefix "environments/development"

# Incident response scans
.\certscanner-windows-amd64.exe -mode local -scanfilesystem -scanmemory `
  -tags "incident-response,forensics" `
  -upload-s3 -s3bucket "security-incidents" `
  -s3prefix "forensics/crypto-analysis"
# Production environment scans
./certscanner-linux-x64 -host prod-servers.txt -cipherscan \
  -tags "production,compliance" \
  -upload-s3 -s3bucket "certscanner-reports" \
  -s3prefix "environments/production"

# Development environment scans  
./certscanner-linux-x64 -host dev-servers.txt -quickscan \
  -tags "development,testing" \
  -upload-s3 -s3bucket "certscanner-reports" \
  -s3prefix "environments/development"

# Incident response scans
./certscanner-linux-x64 -mode local -scanfilesystem -scanmemory \
  -tags "incident-response,forensics" \
  -upload-s3 -s3bucket "security-incidents" \
  -s3prefix "forensics/crypto-analysis"
# Production environment scans
./certscanner-darwin-amd64 -host prod-servers.txt -cipherscan \
  -tags "production,compliance" \
  -upload-s3 -s3bucket "certscanner-reports" \
  -s3prefix "environments/production"

# Development environment scans  
./certscanner-darwin-amd64 -host dev-servers.txt -quickscan \
  -tags "development,testing" \
  -upload-s3 -s3bucket "certscanner-reports" \
  -s3prefix "environments/development"

# Incident response scans (no memory scanning on macOS)
./certscanner-darwin-amd64 -mode local -scanfilesystem \
  -tags "incident-response,forensics" \
  -upload-s3 -s3bucket "security-incidents" \
  -s3prefix "forensics/crypto-analysis"

Multi-Format Reports

# Generate CBOM for compliance and upload to S3
.\certscanner-windows-amd64.exe -host critical-infrastructure.txt -cipherscan `
  -outputformat cbom -output compliance-q4-2024.cbom.json `
  -upload-s3 -s3bucket "compliance-reports" `
  -s3prefix "cbom/quarterly"

# Tychon format for threat intelligence platforms
.\certscanner-windows-amd64.exe -host threat-hunting-targets.txt -cipherscan `
  -outputformat tychon -output threat-analysis.tychon.ndjson `
  -upload-s3 -s3bucket "threat-intelligence" `
  -s3prefix "certscanner/investigations"

# HTML report for executive review
.\certscanner-windows-amd64.exe -host executive-dashboard-hosts.txt -cipherscan `
  -outputformat html -output executive-crypto-report.html `
  -upload-s3 -s3bucket "executive-reports" `
  -s3prefix "security/crypto-assessments"
# Generate CBOM for compliance and upload to S3
./certscanner-linux-x64 -host critical-infrastructure.txt -cipherscan \
  -outputformat cbom -output compliance-q4-2024.cbom.json \
  -upload-s3 -s3bucket "compliance-reports" \
  -s3prefix "cbom/quarterly"

# Tychon format for threat intelligence platforms
./certscanner-linux-x64 -host threat-hunting-targets.txt -cipherscan \
  -outputformat tychon -output threat-analysis.tychon.ndjson \
  -upload-s3 -s3bucket "threat-intelligence" \
  -s3prefix "certscanner/investigations"

# HTML report for executive review
./certscanner-linux-x64 -host executive-dashboard-hosts.txt -cipherscan \
  -outputformat html -output executive-crypto-report.html \
  -upload-s3 -s3bucket "executive-reports" \
  -s3prefix "security/crypto-assessments"
# Generate CBOM for compliance and upload to S3
./certscanner-darwin-amd64 -host critical-infrastructure.txt -cipherscan \
  -outputformat cbom -output compliance-q4-2024.cbom.json \
  -upload-s3 -s3bucket "compliance-reports" \
  -s3prefix "cbom/quarterly"

# Tychon format for threat intelligence platforms  
./certscanner-darwin-amd64 -host threat-hunting-targets.txt -cipherscan \
  -outputformat tychon -output threat-analysis.tychon.ndjson \
  -upload-s3 -s3bucket "threat-intelligence" \
  -s3prefix "certscanner/investigations"

# HTML report for executive review
./certscanner-darwin-amd64 -host executive-dashboard-hosts.txt -cipherscan \
  -outputformat html -output executive-crypto-report.html \
  -upload-s3 -s3bucket "executive-reports" \
  -s3prefix "security/crypto-assessments"

Automated S3 Integration

Scheduled Scanning with S3 Upload

Crontab Configuration

# Daily comprehensive network scan at 2 AM
0 2 * * * /opt/certscanner/certscanner -host @/etc/certscanner/production-hosts.txt \
  -cipherscan -tags "automated,daily,production" \
  -outputformat tychon -output daily-crypto-scan-$(date +\%Y\%m\%d).tychon.ndjson \
  -upload-s3 -s3bucket "certscanner-daily-reports" \
  -s3prefix "production/daily" -s3region "us-east-1"

# Weekly comprehensive local system audit at 3 AM Sunday
0 3 * * 0 /opt/certscanner/certscanner -mode local \
  -scanfilesystem -scanconnected -scanmemory -scanoutlookarchives \
  -tags "automated,weekly,local-audit" \
  -outputformat cbom -output weekly-system-audit-$(date +\%Y\%m\%d).cbom.json \
  -upload-s3 -s3bucket "certscanner-compliance" \
  -s3prefix "system-audits/weekly" -s3region "us-east-1"

# Hourly ARP network discovery  
0 * * * * /opt/certscanner/certscanner -arpscan -quickscan \
  -tags "automated,hourly,network-discovery" \
  -outputformat flatndjson -output arp-discovery-$(date +\%Y\%m\%d-\%H).ndjson \
  -upload-s3 -s3bucket "certscanner-network-discovery" \
  -s3prefix "arp-scans/$(date +\%Y/\%m)" -s3region "us-east-1"

Systemd Service Integration

# /etc/systemd/system/certscanner-s3.service
[Unit]
Description=TYCHON Quantum Readiness S3 Upload Service
After=network.target

[Service]
Type=oneshot
User=certscanner
Group=certscanner
Environment=AWS_PROFILE=certscanner
Environment=AWS_DEFAULT_REGION=us-east-1
ExecStart=/opt/certscanner/certscanner \
  -mode local -scanfilesystem -scanconnected \
  -outputformat flatndjson \
  -upload-s3 -s3bucket "certscanner-reports" \
  -s3prefix "hosts/%H" -s3region "us-east-1"
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/certscanner-s3.timer  
[Unit]
Description=Run TYCHON Quantum Readiness S3 Upload every 6 hours
Requires=certscanner-s3.service

[Timer]
OnCalendar=*-*-* 00,06,12,18:00:00
Persistent=true

[Install]
WantedBy=timers.target

AWS Lambda Processing Pipeline

S3 Event Trigger Lambda Function

import json
import boto3
from datetime import datetime

def lambda_handler(event, context):
    """
    Process TYCHON Quantum Readiness reports uploaded to S3
    Triggered by S3 PutObject events
    """
    
    s3_client = boto3.client('s3')
    
    for record in event['Records']:
        bucket = record['s3']['bucket']['name']
        key = record['s3']['object']['key']
        
        print(f"Processing TYCHON Quantum Readiness report: s3://{bucket}/{key}")
        
        # Download and parse the report
        response = s3_client.get_object(Bucket=bucket, Key=key)
        report_data = response['Body'].read().decode('utf-8')
        
        # Parse based on file extension
        if key.endswith('.ndjson'):
            # Process NDJSON line by line
            for line in report_data.strip().split('\n'):
                if line.strip():
                    record = json.loads(line)
                    process_crypto_asset(record)
        elif key.endswith('.json'):
            # Process complete JSON report
            report = json.loads(report_data)
            process_scan_report(report)
            
        # Optional: Forward to other AWS services
        # - Send to CloudWatch for monitoring
        # - Push to SNS for notifications  
        # - Store in DynamoDB for querying
        # - Trigger Step Functions workflow
        
    return {
        'statusCode': 200,
        'body': json.dumps('Successfully processed TYCHON Quantum Readiness reports')
    }

def process_crypto_asset(asset):
    """Process individual crypto asset from NDJSON"""
    # Extract key security indicators
    if asset.get('certificate_status') == 'expired':
        send_alert('Certificate expired', asset)
    
    if asset.get('pqc_vulnerable') == True:
        send_pqc_alert('PQC vulnerable cipher detected', asset)
        
    if asset.get('security_score', 100) < 50:
        send_alert('Low security score detected', asset)

def process_scan_report(report):
    """Process complete scan report"""
    scan_metadata = report.get('scan_metadata', {})
    
    # Store scan summary in DynamoDB
    dynamodb = boto3.resource('dynamodb')
    table = dynamodb.Table('certscanner-scan-history')
    
    table.put_item(
        Item={
            'scan_id': scan_metadata.get('scan_id'),
            'timestamp': datetime.now().isoformat(),
            'host_count': len(report.get('scan_results', [])),
            'certificate_count': sum(len(host.get('certificates', [])) 
                                   for host in report.get('scan_results', [])),
            's3_location': f"s3://{bucket}/{key}"
        }
    )

Integration Patterns

Real-time Analytics

S3 → Kinesis → Analytics

# Stream NDJSON to Kinesis
aws s3 cp s3://bucket/file.ndjson - | \
aws kinesis put-records --stream-name crypto-assets

S3 → Athena Queries

-- Query crypto assets from S3
SELECT cert_subject_cn, security_score, 
       certificate_days_until_expiry
FROM s3_certscanner_reports 
WHERE security_score < 50;

Compliance & Archival

Quarterly Compliance Reports

# Generate quarterly CBOM
./certscanner -host @critical-systems.txt \
  -cipherscan -outputformat cbom \
  -tags "compliance,q4-2024" \
  -upload-s3 -s3bucket "compliance-archive" \
  -s3prefix "quarterly/2024/q4"

Incident Response Archive

# Forensic crypto analysis
./certscanner -mode local -scanmemory \
  -tags "incident-12345,forensics" \
  -upload-s3 -s3bucket "incident-response" \
  -s3prefix "cases/2024/incident-12345"

Best Practices

Security Considerations

  • • IAM Principle of Least Privilege: Grant only S3 PutObject permissions
  • • Bucket Encryption: Enable server-side encryption (AES-256 or KMS)
  • • Access Logging: Enable S3 access logging for audit trails
  • • Credential Rotation: Regularly rotate AWS access keys
  • • Network Security: Use VPC endpoints for private S3 access
  • • Data Classification: Tag objects with appropriate security classifications

Operational Excellence

  • • Naming Convention: Use consistent S3 key prefixes by environment/function
  • • Lifecycle Management: Configure automated transitions to cheaper storage
  • • Monitoring: Set up CloudWatch alarms for failed uploads
  • • Cost Optimization: Use S3 Intelligent Tiering for cost efficiency
  • • Backup Strategy: Enable Cross-Region Replication for critical reports
  • • Retention Policy: Define clear data retention and deletion policies

Automatic File Organization

TYCHON Quantum Readiness automatically organizes uploads for scalability:

s3://bucket/prefix/YYYY-MM-DD/HH-MM-SS/hosts/scanning-host/targets/target-hosts/filename

Examples:
cert-scanner/
├── production/
│   ├── 2024-12-02/
│   │   ├── 14-30-45/hosts/scanner-01/targets/web-cluster/daily-scan.json
│   │   ├── 14-35-12/hosts/scanner-02/targets/db-cluster/daily-scan.json
│   │   └── 15-00-00/hosts/scanner-01/targets/local-scan/system-audit.json
│   └── 2024-12-03/
│       └── 02-00-00/hosts/scanner-03/targets/192.168.1.0-24/network-scan.json
└── development/
    └── 2024-12-02/
        └── 16-45-30/hosts/dev-scanner/targets/test-env/dev-test.json

This structure automatically scales to thousands of endpoints and multiple scanning hosts without folder conflicts.

Integration Architectures

1. Analytics Pipeline

S3 → Lambda → DynamoDB/RDS → QuickSight Dashboards

2. SIEM Integration

S3 → Lambda → SNS → Security Hub → Third-party SIEM

3. Compliance Workflow

S3 → EventBridge → Step Functions → Compliance Validation

4. Data Lake Architecture

S3 → Glue ETL → Athena → Business Intelligence Tools

Troubleshooting

Common Issues

Access Denied Errors

# Verify credentials
aws sts get-caller-identity

# Test S3 access
aws s3 ls s3://your-bucket-name/

# Check IAM policy attachments
aws iam list-attached-user-policies --user-name certscanner-s3-uploader

Upload Failures

  • • Verify bucket exists and is accessible
  • • Check AWS credentials are configured correctly
  • • Ensure S3 region matches bucket region
  • • Validate file permissions and disk space

Performance Issues

  • • Use S3 Transfer Acceleration for distant regions
  • • Consider multipart uploads for large files
  • • Monitor bandwidth usage during uploads
  • • Use compression for large scan reports