SFTP Integration

This integration is ideal for organizations with legacy systems, existing file-based workflows, or those requiring scheduled batch data transfers.

Overview

With the SFTP integration, you can:

  • Securely transfer files using industry-standard SFTP protocol

  • Support multiple formats (CSV, Excel, JSON)

  • Automate file ingestion into ClearGrid’s processing pipeline

  • Maintain existing batch workflows with minimal changes

  • Ensure encrypted, compliant file transmission

Prerequisites

Technical Requirements

  • SFTP client software or automated transfer scripts

  • Network access to ClearGrid’s SFTP server

  • Files formatted as CSV, Excel, or JSON

  • Understanding of SFTP operations and authentication methods

Organizational Requirements

  • Approval for automated file transfers

  • Coordination with IT for firewall/network configuration

  • Data preparation and validation processes in place

  • Compliance with internal data security policies

Setup Process

Step 1: Request Access

  • Contact ClearGrid Support to request SFTP setup

  • Share your organization details, preferred authentication method, and transfer frequency

  • Receive credentials including:

    • Server Address

    • Port (default 22)

    • Username

    • Authentication Method (password or SSH key)

Step 2: Configure Authentication

  • Username/Password (simpler but less secure)

  • SSH Key Pair (recommended)

    • Generate keys

    • Send public key to ClearGrid

    • Securely store and rotate private keys

Step 3: Network Configuration

  • Open firewall for outbound SFTP (port 22)

  • Test connectivity and authentication

  • Validate file transfer capability

Step 4: Directory Structure

ClearGrid provides a simplified directory structure:

/data/  
  ├── IN/   # Upload your files here  
  └── OUT/  # Processed results, errors, and archives

  • IN → Drop new files for processing

  • OUT → ClearGrid places results, error reports, and archive files

Supported File Formats

CSV

  • UTF-8 encoding, comma-delimited

  • First row = headers

  • Consistent data types

  • Example:

account_id,first_name,last_name,outstanding_balance,phone_number,email_address,due_date,loan_type  
ACC001,John,Smith,5000.00,+1234567890,[email protected],15/06/2025,Personal  
ACC002,Jane,Doe,3500.50,+1234567891,[email protected],20/06/2025,Auto

Excel (.xlsx preferred)

  • Data in the first worksheet

  • Column headers in the first row

  • No merged cells or complex formatting

JSON

  • Valid JSON array of objects

  • Example:

[
  {
    "account_id": "ACC001",
    "first_name": "John",
    "last_name": "Smith",
    "outstanding_balance": 5000.00,
    "phone_number": "+1234567890",
    "email_address": "[email protected]",
    "due_date": "15/06/2025",
    "loan_type": "Personal"
  },
  {
    "account_id": "ACC002",
    "first_name": "Jane",
    "last_name": "Doe",
    "outstanding_balance": 3500.50,
    "phone_number": "+1234567891",
    "email_address": "[email protected]",
    "due_date": "20/06/2025",
    "loan_type": "Auto"
  }
]

Data Requirements

Required Fields

  • account_id (unique identifier)

  • first_name, last_name

  • outstanding_balance

  • Contact information (phone or email)

Recommended Fields

  • Phone number with country code

  • Email address

  • Date of birth (DD/MM/YYYY)

  • National ID

  • Loan type/debt category

  • Due date

  • Original balance

File Processing Workflow

  1. Prepare: Validate and format your file according to requirements

  2. Upload: Place the file into /data/IN/

  3. Processing: ClearGrid ingests the file and validates structure & data

  4. Results: Processed outputs, error reports, and archives are available in /data/OUT/

File Naming Best Practice

  • Use descriptive filenames with timestamps

  • Example: accounts_20250825_batch001.csv

Security

  • Encryption: All data is encrypted in transit via SSH/SFTP

  • Authentication: Password or SSH key-based login

  • Access Control: Isolated org-specific directories, least-privilege access

  • Logging: Full audit trail of transfers and file access

  • Data Retention: Configurable policies with secure deletion and backup

Automation & Scheduling

  • Use cron jobs, Windows Task Scheduler, or third-party tools to automate uploads

  • Processing schedules: Daily, weekly, multiple daily, or on-demand

  • ClearGrid provides monitoring, error notifications, and retry logic

Monitoring & Reporting

  • Real-time: Track file status and errors

  • Historical Reports: View processing success rates and trends

  • Notifications: Email or dashboard alerts on completion/failure

  • API Access: Programmatic status checks

Best Practices

  • Validate data before upload

  • Maintain consistent file naming conventions

  • Use SSH key authentication where possible

  • Regularly review access permissions

  • Schedule large transfers during off-peak hours

Troubleshooting

Common Issues

  • Authentication failures → check credentials or keys

  • Permission denied → verify directory access

  • Data errors → review error reports in /data/OUT/

Migration & Scaling

  • Support for multiple files and parallel uploads

  • Ability to scale to higher volumes as needed

  • Optional migration of historical data via SFTP

  • Integration with automation workflows and APIs

PreviousGoogle Sheet IntegrationNextSecurity Overview

Last updated