Dashcam for events Service
Service Overview
The Dashcam for events Service is designed to automate the retrieval of video footage from dashcams via the Flespi platform. Its primary function is to:
- Request specific video segments from a device based on a trigger (e.g., a harsh braking event).
- Extract the video file from Flespi once available.
- Upload the video to a secure hosting provider (Uploadcare).
- Update the relevant Event/Asset in the Skyledge platform with the video attachment.
This service acts as the bridge between the raw IoT device data (Flespi) and the user-facing platform, ensuring that critical video evidence is reliably captured and accessible.
Key Considerations
Why AWS Serverless?
- Event-Driven: The workflow is inherently event-driven (Fleets System —> SNS —> SQS/Lambda —> Flespi Webhook —> Upload). AWS Lambda is a perfect fit for this asynchronous pattern.
- Scalability: The service needs to handle bursts of requests (e.g., fleet-wide incidents) without manual scaling.
- Cost Efficiency: We only pay for the compute time used during video processing, rather than maintaining idle servers.
- Separation of Concerns: We separate the immutable request log from the mutable processing state using a two-table architecture.
Why Uploadcare?
- Offloaded Processing: Uploadcare handles video ingestion, storage, and delivery (CDN), reducing load on our infrastructure.
- Security: It supports secure, signed uploads (
from_url), ensuring that only authorized services can upload content. - Simplicity: Its API simplifies the complex process of fetching a file from a URL and storing it reliably.
Design Decisions
1. Two-Table Architecture & Deduplication
To ensure data integrity and traceability, we use two separate DynamoDB tables:
- Request Table: A log of all incoming requests where the API call was made, indexed by a deterministic hash of the request parameters. This allows us to deduplicate repeated requests and maintain a solid audit trail.
- Process Table: A mutable state table that tracks the progress of the video upload (Uploadcare token, status). This separation ensures that we can always resume processing without corrupting the original request record.
2. Unique Correlation ID
To handle high concurrency, every unique video request is assigned a unique UUID (correlation_id). This ID is
embedded in the Flespi command metadata. When Flespi sends a webhook (success or failure), it echoes this ID, allowing
us to strictly match the response to the correct record in our database.
3. Strict Correlation & Security
The service implements Strict Correlation. It ignores any incoming Flespi webhook that does not contain a
correlation_id matching a pending request in our database. This prevents “ghost” updates from manual tests or
unrelated device activity.
4. Robustness & Resume Logic
Video uploads can take time. To handle AWS Lambda timeouts:
- We store the Upload Token in the
ProcessTableimmediately upon initiating an upload. - If the function times out and retries, it detects the existing token and resumes polling instead of restarting the upload. This ensures idempotency and reliability.
Data Model
The service uses two Amazon DynamoDB tables.
1. DashCamVideoRequestTable (Immutable Log)
Stores the initial request details. Used for deduplication and audit history.
| Attribute | Type | Description |
|---|---|---|
request_hash |
String | Partition Key. SHA-256 hash of (DeviceID, StartTime, Duration, EventID). |
correlation_id |
String | GSI PK. Unique UUID for Flespi webhook matching. |
device_id |
String | GSI PK. For querying all requests by a specific device. |
status |
String | VIDEO_REQUESTED or FAILED. |
retry_count |
Number | Number of retries attempted. |
ttl |
Number | Expiration timestamp (6 months). |
2. DashCamVideoProcessTable (Mutable State)
Tracks the upload progress.
| Attribute | Type | Description |
|---|---|---|
correlation_id |
String | Partition Key. Links back to the Request Table. |
upload_token |
String | Uploadcare token for resume logic. |
file_id |
String | Final Uploadcare File UUID. |
status |
String | PROCESSING, COMPLETED, FAILED. |
ttl |
Number | Expiration timestamp (6 months). |
Architecture
sequenceDiagram
participant Fleets as Fleet Tracking
participant SNS as Global Event Bus (SNS Topic)
participant q1 as Video Request Queue
participant f1 as Video Request Function
participant RequestTable as Request Table (Log)
participant Flespi
participant api1 as Success Endpoint
participant q2 as Video Process Queue
participant f2 as Video Process Function
participant ProcessTable as Process Table (State)
participant Uploadcare
participant Skyledge
participant api2 as Failure Endpoint
participant q3 as Video Failure Queue
participant f3 as Video Failure Function
participant q4 as DLQ
Fleets ->> SNS: Publish All Events
SNS -->> q1: Publish Event (filter attribute matching)
q1 -->> f1: Event Trigger
f1 ->> f1: Generate Hash & Adjust Start Time (-5s)
f1 ->> RequestTable: Check Hash (Dedup)
alt Duplicate Request
f1 -->> f1: Request Ignored (Duplicate)
else New or Retry
f1 ->> RequestTable: Save Request (deviceId, eventId, VIDEO_REQUESTED)
f1 ->> Flespi: Send Command (with Correlation ID)
end
Note over Flespi: Device uploads video...
Note over Flespi: Webhook for success and failure
Flespi ->> api1: Webhook (Video Ready)
api1 ->> q2: Publish Event
q2 -->> f2: Event Trigger
f2 ->> ProcessTable: Check for Existing State
alt Resume Upload
ProcessTable -->> f2: Return Upload Token
else New Upload
f2 ->> RequestTable: Fetch Request Data
f2 ->> Uploadcare: Initiate Upload
f2 ->> ProcessTable: Save Upload Token
end
f2 ->> Uploadcare: Poll Until Complete
Uploadcare -->> f2: File Ready
f2 ->> Skyledge: Attach Video to Asset
f2 ->> ProcessTable: Update Status (COMPLETED)
Flespi ->> api2: Webhook (Video time out or failure)
api2 ->> q3: Publish Event
q3 -->> f3: Event Trigger
f3 ->> RequestTable: Check for Existing State
alt handle retry, if retry count is under 3
f3 ->> RequestTable: Mark as Failed
f3 ->> q1: Publish Event (retry)
else Send to DLQ (FAILED)
f3 ->> q4: FAILED to request video
end
System Flowchart
Visualizing the complete end-to-end data flow through the infrastructure.
flowchart TD
Fleet[Fleet Tracking System]
SNS[SNS Topic: Global Bus]
q1[SQS: VideoRequestQueue]
f1[Lambda: RequestHandler]
ReqDB[(DynamoDB: RequestTable)]
Flespi((Flespi Platform))
WebhookSuccess[API: Success Webhook]
q2[SQS: VideoNotifyQueue]
f2[Lambda: ProcessHandler]
ProcDB[(DynamoDB: ProcessTable)]
UC((Uploadcare))
Skyledge((Skyledge Platform))
WebhookFail[API: Failure Webhook]
q3[SQS: VideoFailureQueue]
f3[Lambda: RetryHandler]
DLQ[SQS: Dead Letter Queue]
Fleet -->|Publish Event| SNS
SNS -->|Filter & Call| q1
subgraph Request Flow
q1 -->|Trigger| f1
f1 -->|1. Check Hash & Status| ReqDB
f1 -->|2. Send Command| Flespi
f1 -->|3. Save State: VIDEO_REQUESTED| ReqDB
end
subgraph Process Flow
Flespi -.->|Video Ready| WebhookSuccess
WebhookSuccess --> q2
q2 -->|Trigger| f2
f2 -->|1. Check/Resume| ProcDB
f2 -->|2. Upload URL & receive token| UC
f2 -->|3. Add entry with UC token| ProcDB
f2 -.->|File Ready Pooling| UC
f2 -->|4. Update State: COMPLETED| ProcDB
f2 -->|5. Update Asset| Skyledge
end
subgraph Retry Flow
Flespi -.->|Error/Timeout| WebhookFail
WebhookFail --> q3
q3 -->|Trigger| f3
f3 -->|1. Mark FAILED| ReqDB
f3 -->|2a. Retry < 3| q1
f3 -->|2b. Retry >= 3| DLQ
end
classDef aws fill:#FF9900,stroke:#232F3E,color:white;
classDef ext fill:#232F3E,stroke:white,color:white;
classDef db fill:#3F8624,stroke:#232F3E,color:white;
class q1,q2,q3,DLQ,SNS,f1,f2,f3 aws;
class Fleet,Flespi,UC,Skyledge ext;
class ReqDB,ProcDB db;
Component Details
For detailed implementation logic of the Request, Process, and Retry handlers, please refer to the Component Documentation.