Skip to content

Dashcam for events Service

Service Overview

The Dashcam for events Service is designed to automate the retrieval of video footage from dashcams via the Flespi platform. Its primary function is to:

  1. Request specific video segments from a device based on a trigger (e.g., a harsh braking event).
  2. Extract the video file from Flespi once available.
  3. Upload the video to a secure hosting provider (Uploadcare).
  4. Update the relevant Event/Asset in the Skyledge platform with the video attachment.

This service acts as the bridge between the raw IoT device data (Flespi) and the user-facing platform, ensuring that critical video evidence is reliably captured and accessible.

Key Considerations

Why AWS Serverless?

  • Event-Driven: The workflow is inherently event-driven (Fleets System —> SNS —> SQS/Lambda —> Flespi Webhook —> Upload). AWS Lambda is a perfect fit for this asynchronous pattern.
  • Scalability: The service needs to handle bursts of requests (e.g., fleet-wide incidents) without manual scaling.
  • Cost Efficiency: We only pay for the compute time used during video processing, rather than maintaining idle servers.
  • Separation of Concerns: We separate the immutable request log from the mutable processing state using a two-table architecture.

Why Uploadcare?

  • Offloaded Processing: Uploadcare handles video ingestion, storage, and delivery (CDN), reducing load on our infrastructure.
  • Security: It supports secure, signed uploads (from_url), ensuring that only authorized services can upload content.
  • Simplicity: Its API simplifies the complex process of fetching a file from a URL and storing it reliably.

Design Decisions

1. Two-Table Architecture & Deduplication

To ensure data integrity and traceability, we use two separate DynamoDB tables:

  • Request Table: A log of all incoming requests where the API call was made, indexed by a deterministic hash of the request parameters. This allows us to deduplicate repeated requests and maintain a solid audit trail.
  • Process Table: A mutable state table that tracks the progress of the video upload (Uploadcare token, status). This separation ensures that we can always resume processing without corrupting the original request record.

2. Unique Correlation ID

To handle high concurrency, every unique video request is assigned a unique UUID (correlation_id). This ID is embedded in the Flespi command metadata. When Flespi sends a webhook (success or failure), it echoes this ID, allowing us to strictly match the response to the correct record in our database.

3. Strict Correlation & Security

The service implements Strict Correlation. It ignores any incoming Flespi webhook that does not contain a correlation_id matching a pending request in our database. This prevents “ghost” updates from manual tests or unrelated device activity.

4. Robustness & Resume Logic

Video uploads can take time. To handle AWS Lambda timeouts:

  • We store the Upload Token in the ProcessTable immediately upon initiating an upload.
  • If the function times out and retries, it detects the existing token and resumes polling instead of restarting the upload. This ensures idempotency and reliability.

Data Model

The service uses two Amazon DynamoDB tables.

1. DashCamVideoRequestTable (Immutable Log)

Stores the initial request details. Used for deduplication and audit history.

Attribute Type Description
request_hash String Partition Key. SHA-256 hash of (DeviceID, StartTime, Duration, EventID).
correlation_id String GSI PK. Unique UUID for Flespi webhook matching.
device_id String GSI PK. For querying all requests by a specific device.
status String VIDEO_REQUESTED or FAILED.
retry_count Number Number of retries attempted.
ttl Number Expiration timestamp (6 months).

2. DashCamVideoProcessTable (Mutable State)

Tracks the upload progress.

Attribute Type Description
correlation_id String Partition Key. Links back to the Request Table.
upload_token String Uploadcare token for resume logic.
file_id String Final Uploadcare File UUID.
status String PROCESSING, COMPLETED, FAILED.
ttl Number Expiration timestamp (6 months).

Architecture

sequenceDiagram
    participant Fleets as Fleet Tracking
  participant SNS as Global Event Bus (SNS Topic)
  participant q1 as Video Request Queue
  participant f1 as Video Request Function
  participant RequestTable as Request Table (Log)
    participant Flespi
  participant api1 as Success Endpoint
  participant q2 as Video Process Queue
  participant f2 as Video Process Function
  participant ProcessTable as Process Table (State)
  participant Uploadcare
  participant Skyledge
  participant api2 as Failure Endpoint
  participant q3 as Video Failure Queue
  participant f3 as Video Failure Function
  participant q4 as DLQ
  Fleets ->> SNS: Publish All Events
  SNS -->> q1: Publish Event (filter attribute matching)
  q1 -->> f1: Event Trigger
  f1 ->> f1: Generate Hash & Adjust Start Time (-5s)
  f1 ->> RequestTable: Check Hash (Dedup)

  alt Duplicate Request
    f1 -->> f1: Request Ignored (Duplicate)
  else New or Retry
    f1 ->> RequestTable: Save Request (deviceId, eventId, VIDEO_REQUESTED)
    f1 ->> Flespi: Send Command (with Correlation ID)
  end

  Note over Flespi: Device uploads video...
    Note over Flespi: Webhook for success and failure
  Flespi ->> api1: Webhook (Video Ready)
  api1 ->> q2: Publish Event
  q2 -->> f2: Event Trigger
  f2 ->> ProcessTable: Check for Existing State

  alt Resume Upload
    ProcessTable -->> f2: Return Upload Token
  else New Upload
    f2 ->> RequestTable: Fetch Request Data
    f2 ->> Uploadcare: Initiate Upload
    f2 ->> ProcessTable: Save Upload Token
  end

  f2 ->> Uploadcare: Poll Until Complete
  Uploadcare -->> f2: File Ready
  f2 ->> Skyledge: Attach Video to Asset
  f2 ->> ProcessTable: Update Status (COMPLETED)
  Flespi ->> api2: Webhook (Video time out or failure)
  api2 ->> q3: Publish Event
  q3 -->> f3: Event Trigger
  f3 ->> RequestTable: Check for Existing State
  alt handle retry, if retry count is under 3
    f3 ->> RequestTable: Mark as Failed
    f3 ->> q1: Publish Event (retry)
  else Send to DLQ (FAILED)
    f3 ->> q4: FAILED to request video
  end

System Flowchart

Visualizing the complete end-to-end data flow through the infrastructure.

flowchart TD
    Fleet[Fleet Tracking System]
    SNS[SNS Topic: Global Bus]
    q1[SQS: VideoRequestQueue]
    f1[Lambda: RequestHandler]
    ReqDB[(DynamoDB: RequestTable)]
    Flespi((Flespi Platform))
    
    WebhookSuccess[API: Success Webhook]
    q2[SQS: VideoNotifyQueue]
    f2[Lambda: ProcessHandler]
    ProcDB[(DynamoDB: ProcessTable)]
    UC((Uploadcare))
    Skyledge((Skyledge Platform))
    
    WebhookFail[API: Failure Webhook]
    q3[SQS: VideoFailureQueue]
    f3[Lambda: RetryHandler]
    DLQ[SQS: Dead Letter Queue]


    Fleet -->|Publish Event| SNS
    SNS -->|Filter & Call| q1

    subgraph Request Flow
        q1 -->|Trigger| f1
        f1 -->|1. Check Hash & Status| ReqDB
        f1 -->|2. Send Command| Flespi
        f1 -->|3. Save State: VIDEO_REQUESTED| ReqDB
    end

    subgraph Process Flow
        Flespi -.->|Video Ready| WebhookSuccess
        WebhookSuccess --> q2
        q2 -->|Trigger| f2
        f2 -->|1. Check/Resume| ProcDB
        f2 -->|2. Upload URL & receive token| UC
        f2 -->|3. Add entry with UC token| ProcDB
        f2 -.->|File Ready Pooling| UC
        f2 -->|4. Update State: COMPLETED| ProcDB
        f2 -->|5. Update Asset| Skyledge
    end

    subgraph Retry Flow
        Flespi -.->|Error/Timeout| WebhookFail
        WebhookFail --> q3
        q3 -->|Trigger| f3
        f3 -->|1. Mark FAILED| ReqDB
        f3 -->|2a. Retry < 3| q1
        f3 -->|2b. Retry >= 3| DLQ
    end
    
    classDef aws fill:#FF9900,stroke:#232F3E,color:white;
    classDef ext fill:#232F3E,stroke:white,color:white;
    classDef db fill:#3F8624,stroke:#232F3E,color:white;
    
    class q1,q2,q3,DLQ,SNS,f1,f2,f3 aws;
    class Fleet,Flespi,UC,Skyledge ext;
    class ReqDB,ProcDB db;

Component Details

For detailed implementation logic of the Request, Process, and Retry handlers, please refer to the Component Documentation.