File System Operations

---
name: file-system-operations
description: "Safely perform file system operations with atomic writes, locking, and cross-platform path handling for robust autonomous agents."
metadata:
  tags: "autonomous-agents, file-system-operations, atomic-writes, file-locking, temp-files, path-normalization, cross-platform"
  source: "https://skilldb.dev/skills/autonomous-agent-skills/file-system-operations"
  pack: "autonomous-agent-skills"
  category: "Autonomous Agents"
---

# File System Operations

## When to use this skill
Use when the user says things like:
- "I need to atomically write a file."
- "Lock this file during updates."
- "Normalize a user-provided path to prevent traversal."
- "Use a temporary file before replacing the target."


You are an autonomous agent that interacts with file systems as a core part of your work.
Treat file operations as potentially destructive actions that demand the same care as database writes.
Every read or write should be intentional, safe, and resilient to failure.

## Philosophy

File systems are shared, mutable state.
Other processes, users, or even the OS itself may modify files concurrently.
Your operations must account for partial failures, race conditions, and platform differences.
Prefer operations that are atomic, idempotent, and reversible.
When in doubt, err on the side of safety — a failed write that leaves the original intact is always preferable to a partial write that corrupts data.

## Techniques

### Atomic Writes
Never write directly to a target file.
Instead, write to a temporary file in the same directory, then rename it to the final path.
Renaming within the same filesystem is atomic on both POSIX and Windows (with caveats).
This prevents readers from seeing half-written content.
The pattern is: open temp file, write all content, flush, fsync, close, then rename.
If any step fails, the original file remains untouched.

### Temp File Patterns
Use the OS-provided temp directory for scratch work, but place temp files destined for atomic rename in the same directory as the target — cross-device renames are not atomic and may fail.
Use unique suffixes (PID, timestamp, random string) to avoid collisions.
Always clean up temp files in a finally block or equivalent error-handling construct.
Consider using `mkstemp` or equivalent to create temp files with restricted permissions, preventing other users from reading sensitive intermediate data.

### File Locking
Use advisory locks (flock on POSIX, LockFileEx on Windows) when multiple processes may write to the same file.
Hold locks for the shortest duration possible.
Never assume a lock file's existence means the lock is held — the owning process may have crashed.
Implement lock timeouts to avoid deadlocks.
When using lock files (a separate `.lock` file), create them atomically with `O_CREAT | O_EXCL` and remove them in a finally block.

### Directory Traversal Safety
When constructing paths from user input, always resolve and normalize the path first, then verify it stays within an expected base directory.
Reject paths containing `..` segments that escape the sandbox.
Use `path.resolve()` or `os.path.realpath()` before comparison.
Be aware that on case-insensitive filesystems, path comparison must also be case-insensitive to prevent bypass via mixed-case segments.

### Path Normalization
Always normalize paths before comparing or storing them.
Convert backslashes to forward slashes when working cross-platform.
Use `path.join()` or equivalent rather than string concatenation.
Never hardcode path separators.
Be aware of trailing separator differences: `dir/` and `dir` may or may not refer to the same thing depending on the operation and platform.

### Symlink Awareness
Before operating on a file, decide whether you intend to operate on the symlink or its target.
Use `lstat` vs `stat` accordingly.
When deleting directories recursively, be cautious of symlinks that point outside the tree — do not follow them blindly.
When creating files, check whether the target path is a symlink that points somewhere unexpected.
Symlink resolution can differ between platforms, especially on Windows where symlinks require elevated privileges by default.

### Large File Handling
Stream large files rather than reading them entirely into memory.
Use chunked reads with a fixed buffer size (typically 64KB to 1MB).
For line-by-line processing, use readline interfaces or generators.
When reporting progress, track bytes processed against total file size.
For files that may grow while being read (log files), use a snapshot approach or handle the moving target explicitly.
Consider memory-mapped files for random access patterns on large files.

### File Watching
Use OS-native file watching (inotify, FSEvents, ReadDirectoryChangesW) rather than polling.
Debounce rapid successive events — a save operation may trigger multiple events.
Handle the case where a watched directory is deleted or renamed.
Always provide a fallback polling mode for network-mounted filesystems where native events may not fire.
Be aware of platform-specific event limits (inotify has a configurable maximum number of watches).

### Cross-Platform Path Handling
Forward slashes work on Windows in most contexts, but not all (notably some Windows API calls and shell commands).
Use `path.sep` when you need the platform-native separator.
Treat paths as case-insensitive on Windows and macOS (by default), case-sensitive on Linux.
Be aware of reserved filenames on Windows (CON, PRN, NUL, COM1-9, LPT1-9) and maximum path length limitations (260 characters by default, though long paths can be enabled).

### Encoding and Line Endings
Always be explicit about text encoding when reading or writing files.
Default to UTF-8 with BOM detection.
Handle BOM (byte order mark) gracefully — strip it when reading, do not add it when writing unless specifically required.
Normalize line endings consistently: use `\
` internally, convert to platform convention only at output boundaries.

## Best Practices

- Check file existence only as an optimization hint, never as a security gate. The file can appear or disappear between check and use (TOCTOU race).
- Set appropriate file permissions at creation time, not after. On POSIX, use umask-aware creation modes.
- Use `O_CREAT | O_EXCL` (or equivalent) when you need to guarantee you are creating a new file, not overwriting an existing one.
- Flush and fsync before closing files when durability matters. A successful `write()` does not mean the data has reached disk.
- Handle `ENOSPC`, `EACCES`, `EMFILE`, and `ENAMETOOLONG` gracefully with clear error messages rather than generic failure.
- When reading configuration files, provide sensible defaults if the file does not exist. Distinguish between "missing file" and "empty file."
- Prefer relative paths in configuration and output so artifacts remain portable across machines.
- Log the full resolved path in error messages to aid debugging.
- When creating directory trees, use `mkdir -p` (recursive creation) but verify the final permissions are correct.
- Before writing, verify the target directory exists. Create it if needed, but never silently create deeply nested structures that may indicate a misconfigured path.
- Use exclusive file creation when generating unique output files (reports, exports) to prevent accidental overwrites.
- Close files in finally blocks or use with-statement / using patterns. Leaked descriptors accumulate and eventually hit the per-process limit.

## Anti-Patterns

- **Writing directly to the target file** — Readers may see partial content; crashes leave corrupted files. Always use atomic write-then-rename.
- **Using string concatenation for paths** — Leads to double separators, missing separators, and platform bugs. Use path.join or equivalent.
- **Catching and ignoring all file errors** — Different errors demand different responses. ENOENT may be expected; EACCES is a configuration problem; ENOSPC needs operator intervention.
- **Recursive deletion without symlink checks** — A symlink pointing to `/` inside your tree could cause catastrophic data loss if you follow it during recursive delete.
- **Hardcoding `/tmp`** — Use `os.tmpdir()` or equivalent. On some systems, `/tmp` is a ramdisk with limited space or different cleanup policies.
- **Reading entire large files into memory** — This causes memory pressure, potential OOM, and slow performance. Stream instead.
- **Polling for file changes at high frequency** — Wastes CPU and battery. Use OS-native watchers with a polling fallback only when necessary.
- **Assuming UTF-8 encoding** — Always be explicit about encoding. Detect or require encoding declaration for text files.
- **Not handling partial writes** — Network filesystems and full disks can cause writes to succeed partially. Always verify written byte counts and use fsync to confirm durability.
- **Leaking file descriptors** — Open files without corresponding closes accumulate and eventually hit the per-process file descriptor limit, causing unrelated operations to fail.