Skip to content

Stock Import - Overview

Purpose

The Stock Import island is a separate Symfony application that handles importing stock data from 50-60 suppliers. It runs on an isolated infrastructure for security reasons (FTP server receiving external files).

Architecture

┌─────────────────────────────────────────────┐
│              STOCK IMPORT                   │
│                                             │
│  ┌─────────────┐    ┌──────────────────┐   │
│  │ FTP Watcher │───►│ File Processor   │   │
│  │  (cron)     │    │                  │   │
│  └─────────────┘    └────────┬─────────┘   │
│                              │              │
│                     ┌────────▼─────────┐   │
│                     │ Article Matching │   │
│                     │     Engine       │   │
│                     └────────┬─────────┘   │
│                              │              │
│                     ┌────────▼─────────┐   │
│                     │  Stock Update    │   │
│                     │    Service       │   │
│                     └────────┬─────────┘   │
│                              │              │
└──────────────────────────────┼──────────────┘
                               │ REST API
                               ▼
                    ┌─────────────────────┐
                    │      WEBSHOP        │
                    └─────────────────────┘

Components

1. FTP Watcher

  • Monitors FTP directory for new files
  • Runs as scheduled cron job
  • Moves processed files to archive

2. File Processor

  • Parses CSV/XML files
  • Supplier-specific format handling
  • Validates file structure

3. Article Matching Engine

  • Matches supplier articles to internal catalog
  • Confidence scoring for uncertain matches
  • Manual review queue for low-confidence matches

4. Stock Update Service

  • Updates stock levels in database
  • Pushes updates to Webshop via API

Security

  • FTP server is isolated from main network
  • Only receives files, no outbound connections
  • Stock Import pulls files, no push access

Data Flow

  1. Supplier uploads CSV/XML to FTP
  2. FTP Watcher detects new file
  3. File Processor parses based on supplier config
  4. Matching Engine maps to internal articles
  5. Stock Update sends to Webshop API
  6. File moved to archive with processing log

Key Improvements

Current New
Matching sometimes incorrect Confidence scoring + review queue
No audit trail Full matching history
Manual investigation Alerting for problems
Single matching attempt Multi-identifier verification

Connector Runtime Contract (AR-004 Closure)

All supplier connectors (FTP/API/CSV/XML) run through one reliability contract.

Idempotency

Each supplier feed message must carry a deterministic idempotency key:

supplier-feed:{supplierId}:{sourceType}:{externalFileNameOrCursor}:{contentHash}

Rules:

  1. Duplicate key means acknowledge and skip side effects.
  2. Key is stored in consumer inbox/dedup table.
  3. Stock write operations must be upsert/idempotent per (supplierId, articleNumber, batchId).

Retry and Backoff

  1. Temporary failures (5xx, connection timeout, transient auth errors) retry with exponential backoff.
  2. Validation failures (4xx malformed payload, schema mismatch) do not blind retry; they are marked failed with operator action required.
  3. Max retry attempts are connector-configurable with safe defaults.

Dead-Letter Queue and Replay

  1. Exhausted retries move message to supplier DLQ.
  2. DLQ records retain payload pointer, error stack, supplier id, and idempotency key.
  3. Replay command must support scoped replay by supplier and date range.

Checkpoints and Watermarks

Keep resumable state per supplier connector:

  • last_successful_file or cursor
  • last_successful_timestamp
  • last_processed_hash
  • last_batch_id

Restart behavior:

  1. Resume from last committed checkpoint.
  2. Re-read overlap window where needed; idempotency keys prevent duplicates.

Queue Model

Queue classes:

  1. high-priority: incremental stock updates and urgent correction batches.
  2. bulk: full feed imports and backfills.

Minimum Operational Metrics

  1. Import success rate by supplier.
  2. Retry count and DLQ depth by supplier.
  3. Processing lag from file arrival to committed stock update.
  4. Duplicate-delivery drop count.

Acceptance Scenarios (Gherkin)

Feature: Supplier connector runtime reliability

  Scenario: Duplicate supplier feed is ignored safely
    Given supplier feed message M with idempotency key K was already processed
    When message M is delivered again
    Then no duplicate stock mutation should be committed
    And the message should be acknowledged as duplicate

  Scenario: Temporary connector failure is retried and succeeds
    Given supplier connector receives a temporary upstream timeout
    When retry policy is applied
    Then the message should be retried with backoff
    And successful processing should commit stock and advance checkpoint

  Scenario: Exhausted retries route to DLQ and can be replayed
    Given supplier feed processing keeps failing with temporary errors
    When max retries are exhausted
    Then the message should be moved to supplier DLQ
    When operator replays the message after fix
    Then processing should succeed without duplicate side effects