Blacklight

Blacklight is a powerful secret, keys and sensitive data scanning tool that helps you detect and prevent sensitive information leaks in your codebase, databases, cloud storage, and communication platforms.

Features

• Multi-Source Scanning

  • Local files and directories
  • Databases (PostgreSQL, MySQL)
  • AWS S3 buckets
  • Slack workspace messages and files
  • Cloud Storage (Google Drive, Dropbox)
  • Git repositories

• Advanced Detection

  • Pattern-based secret detection
  • Context-aware scanning
  • Multi-language support
  • Configurable severity levels
  • Rule categorization
  • Smart file format detection

• Supported File Formats

  • Plain text files
  • JSON files (with nested object support)
  • YAML files (with nested object support)
  • XML files (with attribute scanning)
  • INI/Config files
  • Environment files (.env)
  • Configuration files

• User Experience

  • Cross-platform compatibility (Windows, Linux, macOS)
  • Beautiful table output with go-pretty formatting
  • Color-coded severity indicators
  • Detailed violation reporting
  • Rich context for findings

Installation

Quick Install (Linux/macOS)

# macOS (Apple Silicon)
curl -Lo blacklight https://github.com/adaptive-scale/blacklight/releases/latest/download/blacklight-darwin-arm64 && \
chmod +x blacklight && \
sudo mv blacklight /usr/local/bin/

# macOS (Intel)
curl -Lo blacklight https://github.com/adaptive-scale/blacklight/releases/latest/download/blacklight-darwin-amd64 && \
chmod +x blacklight && \
sudo mv blacklight /usr/local/bin/

# Linux (x86_64)
curl -Lo blacklight https://github.com/adaptive-scale/blacklight/releases/latest/download/blacklight-linux-amd64 && \
chmod +x blacklight && \
sudo mv blacklight /usr/local/bin/

# Linux (ARM64)
curl -Lo blacklight https://github.com/adaptive-scale/blacklight/releases/latest/download/blacklight-linux-arm64 && \
chmod +x blacklight && \
sudo mv blacklight /usr/local/bin/

# Verify installation
blacklight version

Using Go

# Using go install
go install github.com/adaptive-scale/blacklight@latest

# Or clone and build
git clone https://github.com/adaptive-scale/blacklight.git
cd blacklight
make build

Docker

# Pull the latest image
docker pull adaptivescale/blacklight:latest

# Run a scan
docker run -v $(pwd):/workspace adaptivescale/blacklight:latest scan /workspace

Windows

Download the appropriate .exe file from our releases page and add it to your PATH.

Verifying the Installation

After installation, verify that Blacklight is working correctly:

# Check version
blacklight version

# View help
blacklight --help

# Run a test scan
blacklight scan --help

Usage

Basic Usage

# Scan a directory
blacklight scan /path/to/directory

# Scan with verbose output
blacklight scan /path/to/directory --verbose

# Scan a database
blacklight scan --db "postgresql://user:pass@localhost:5432/dbname"

# Scan an S3 bucket
blacklight scan --s3 "s3://bucket-name"

# Scan cloud storage
blacklight scan --drive "gdrive://folder-id"

Rule Management

# List all rules
blacklight rules list

# List rules by type
blacklight rules list --type cloud

# List rules by severity
blacklight rules list --severity 3

# Add a new rule
blacklight rules add --name "Custom API Key" \
                     --regex "api_key_[a-zA-Z0-9]{32}" \
                     --severity 2 \
                     --type "secret"

Slack Workspace Scanning

Blacklight includes a powerful Slack scanner that can detect secrets and sensitive information in:

• Channel messages (public and private)
• Message threads
• Direct messages (DMs)
• Group messages
• Shared files
• File comments

Setup

1. Create a Slack App at https://api.slack.com/apps
2. Add the following OAuth scopes:

   channels:history   - View messages and other content in public channels
   channels:read     - View basic information about public channels
   files:read       - View files shared in channels and conversations
   groups:history   - View messages and other content in private channels
   groups:read      - View basic information about private channels
   im:history      - View messages and other content in direct messages
   im:read        - View basic information about direct messages
   mpim:history   - View messages and other content in group direct messages
   mpim:read     - View basic information about group direct messages
   

3. Install the app to your workspace
4. Copy the Bot User OAuth Token (starts with xoxb-)

Usage

# Basic scan of all accessible channels
blacklight slack --token xoxb-your-token

# Scan specific channels
blacklight slack --token xoxb-your-token --channels C01234567,C89012345

# Scan recent messages
blacklight slack --token xoxb-your-token --days 7

# Full scan including threads and files
blacklight slack --token xoxb-your-token --include-threads --include-files

Configuration Options

Performance Considerations

• File scanning is disabled by default to improve performance
• Files larger than 10MB are skipped
• Use the --days flag to limit the scan window
• Specify channels to scan for faster results

Cloud Storage Scanning

Blacklight can scan files in various cloud storage services for secrets and sensitive information:

Implemented Providers

• Google Drive (gdrive://)

  • Scans files in specified folders
  • Supports file content analysis
  • Respects file size limits
  • OAuth2 authentication
  • Automatic file format detection
  • Recursive folder scanning

• Dropbox (dropbox://)

  • Full folder scanning
  • File content analysis
  • Path-based access
  • Access token authentication
  • Smart file format handling
  • Size-based file filtering

Coming Soon

• OneDrive (onedrive://) - In development
• Box (box://) - Planned

Authentication

Each provider requires appropriate authentication:

# Google Drive - OAuth2 client configuration
export CLOUD_TOKEN='{"client_id":"...","client_secret":"...","redirect_uris":["..."]}'

# Dropbox - Access Token
export CLOUD_TOKEN="your-dropbox-access-token"

Usage Examples

# Scan Google Drive folder
blacklight scan --drive "gdrive://folder-id"

# Scan Dropbox folder
blacklight scan --drive "dropbox://path/to/folder"

# Include shared files (Google Drive)
blacklight scan --drive "gdrive://folder-id" --include-shared

# Limit scan history
blacklight scan --drive "dropbox://folder" --days 7

# Adjust file size limit
blacklight scan --drive "gdrive://folder-id" --max-size 5242880  # 5MB

Configuration Options

File Format Support

The cloud storage scanner automatically detects and processes various file formats:

Performance Considerations

• Files larger than the max-size limit are skipped
• Use --days to limit scan scope
• Specify precise folder paths for faster scans
• Token expiration is handled automatically
• File format detection optimizes scanning

Security Notes

• Tokens should be kept secure and not shared
• Use read-only access tokens when possible
• Consider using environment variables for token storage
• Regularly rotate access tokens
• Ensure proper access permissions

Rule Types

Blacklight organizes its scanning rules into the following categories:

Authentication & Authorization

• auth: Authentication tokens, passwords, OAuth credentials
• key: Cryptographic keys (RSA, DSA, PGP, SSH)

Cloud & Infrastructure

• cloud: Cloud provider credentials (AWS, Azure, GCP)
• container: Container platform secrets (Docker, Kubernetes)
• iac: Infrastructure as Code secrets (Terraform)
• cdn: Content Delivery Network tokens

APIs & Services

• api: Generic and service-specific API keys
• monitoring: Monitoring service tokens (NewRelic, Rollbar)
• ci: CI/CD platform credentials
• vcs: Version Control System tokens (GitHub, GitLab)

Payment & Financial

• payment: Payment gateway credentials
• pci: Payment Card Industry data
• ecommerce: E-commerce platform tokens

Data & Storage

• database: Database credentials and endpoints
• messaging: Message queue credentials
• package: Package registry tokens

Other

• secret: Generic secrets and environment variables
• social: Social media platform tokens
• security: Security-related credentials
• config: Configuration file secrets
• ai: AI service credentials

Rule Configuration

Rules are stored in ~/.blacklight/rules.yaml. Each rule has the following properties:

Example Rule File

- id: "aws_access_key"
  name: "AWS Access Key"
  description: "Amazon Web Services access key ID"
  regex: "AKIA[0-9A-Z]{16}"
  severity: 3
  type: "cloud"
  disabled: false

- id: "stripe_key"
  name: "Stripe API Key"
  description: "Stripe secret API key"
  regex: "sk_live_[0-9a-zA-Z]{24}"
  severity: 3
  type: "payment"
  disabled: false

Output Format

Blacklight provides rich, color-coded output:

[Severity 3]: AWS Access Key Found
Location: slack://channel/C0123456/message/1234567890.123
Context: ...config = { accessKeyId: "AKIAXXXXXXXXXXXXXXXX", region: "us-east-1" }...
Match: AKIAXXXXXXXXXXXXXXXX
--------------------------------------------------------------------------------

The table output uses go-pretty for enhanced readability:

╭──────────────────────────┬──────────┬──────────┬─────────┬───────────────────────────────────╮
│ NAME                     │ TYPE     │ SEVERITY │ STATUS  │ PATTERN                           │
├──────────────────────────┼──────────┼──────────┼─────────┼───────────────────────────────────┤
│ AWS Access Key           │ cloud    │    3     │ Enabled │ AKIA[0-9A-Z]{16}                 │
│ Stripe API Key          │ payment  │    3     │ Enabled │ sk_live_[0-9a-zA-Z]{24}          │
╰──────────────────────────┴──────────┴──────────┴─────────┴───────────────────────────────────╯

Default Rules

Blacklight comes with a comprehensive set of pre-configured rules for detecting various types of secrets and sensitive information:

Authentication & Authorization

Cloud Services

Payment & Financial

Database & Storage

Communication & Messaging

Development & CI/CD

AI & Machine Learning

Analytics & Monitoring

Generic Patterns

Custom Rules

You can add your own custom rules by creating a file at ~/.blacklight/rules.yaml:

- id: "custom_api_key"
  name: "Custom API Key"
  description: "Detects custom API key pattern"
  regex: "myapi_[a-zA-Z0-9]{32}"
  severity: 2
  type: "api"
  disabled: false

- id: "internal_token"
  name: "Internal Service Token"
  description: "Internal service authentication token"
  regex: "int_[a-zA-Z0-9]{24}"
  severity: 3
  type: "auth"
  disabled: false

Rule Properties

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

Copyright © 2025 Debarshi Basak

Licensed under the Apache License, Version 2.0