Coding

---
name: file-processing
description: Stream processing, large file uploads, and IO efficiency.
triggers: [file processing]
tags: [coding]
role: eng-backend
context_cost: low
---
# file-processing Skill

## Goal
This skill guides the handling of binary data and large datasets without blowing up RAM.

## Steps
## 1. Streaming (Chunks)
- **Problem**: Reading 1GB CSV into memory (RAM).
- **Solution**: Stream it line by line.
  - nodejs: `fs.createReadStream().pipe(csvParser())`.
  - python: `pd.read_csv(chunksize=1000)`.

## 2. Uploads (Presigned URLs)
- **Anti-Pattern**: Client -> Server (RAM buffer) -> S3.
- **Pattern**: Client -> (Request URL) -> Server; Server -> (Signed URL) -> Client; Client -> (Direct Upload) -> S3.
- **Benefit**: Server CPU/RAM is zero during upload. Infinite scalability.

## 3. Malware Scanning
- Never trust user files.
- **Sync**: ClamAV scan on upload (slow).
- **Async**: Upload to S3 (Quarantine Bucket) -> Lambda triggers Scan -> Move to Clean Bucket.

## 4. Exporting Large Data
- Don't return 100k rows in JSON response (timeout).
- **Pattern**:
  1. User requests Export.
  2. Server puts job in Queue.
  3. Worker generates CSV, uploads to S3.
  4. Server notifies User with Download Link.

## 5. File Types
- **Magic Bytes**: Check file signature (hex), not just extension.
- **Image Processing**: Resize/Optimize *async*. Use `sharp` (Node) or `Pillow` (Python).

## Security & Guardrails

### 1. Skill Security (File Processing)
- **Zip Bomb / Zip Slip Vulnerabilities**: When processing large files (Step 1), the agent might implement streaming for CSVs but neglect archive extraction risks. If an attacker uploads a highly compressed 10GB file (Zip Bomb) or a file containing relative paths (`../../../etc/passwd`), naive extraction will crash the server or overwrite arbitrary files. The agent must enforce strict decompression ratio limits, maximum file size caps, and path sanitization on all archive streams.
- **Malware Execution via Magic Bytes Parsing**: In Step 5, the agent checks "Magic Bytes" to verify file types. If the agent uses an insecure, outdated library to parse these headers (e.g., a vulnerable image processing binary), simply inspecting the malware to determine its type could result in arbitrary code execution. The agent must mandate the use of hardened, memory-safe inspection binaries deployed in isolated environments (e.g., ephemeral Lambdas) before the main application server ever touches the byte stream.

### 2. System Integration Security
- **Presigned URL Exfiltration**: In Step 2, the pattern dictates "Server -> (Signed URL) -> Client." If the agent implements this by generating overly permissive Signed URLs (e.g., valid for 24 hours, lack of precise file-path bounding, allowing GET/PUT/DELETE instead of just PUT), an attacker can hijack the URL to read/write arbitrary objects in the S3 bucket. The agent must rigorously scope the IAM policy tied to the Presigned URL generation to a single, specific object key with a lifespan `< 15 minutes`.
- **Quarantine Bypass**: Step 3 describes a strict Async Malware Scanning flow. If the agent implements the logic such that the "Clean Bucket" is publicly readable, but the application accidentally serves links directly from the "Quarantine Bucket" before the Lambda finishes, the security control is bypassed. The agent must enforce network-level IAM policies ensuring the application layer *cannot physical read* from the Quarantine Bucket under any circumstances.

### 3. LLM & Agent Guardrails
- **Memory Exhaustion Under Load**: The LLM understands "Streaming" abstractly, but might write code that accumulates stream chunks into an array before processing (pseudo-streaming), defeating the purpose. The agent must statically analyze the generated code to ensure a true $O(1)$ memory pipeline, explicitly failing the implementation if it identifies unbounded array `push()` operations within the data stream handler.
- **Hallucinated Async Safety**: The LLM might assume that because a file export job is queued (Step 4), it is inherently secure against Denial of Service. However, if the attacker can enqueue 100,000 export jobs, they exhaust the queue and bankrupt the AWS bill. The agent must enforce rate-limiting, job deduplication, and resource quotas on all asynchronous file processing triggers generated by this skill.