How We Made SpaceWise 5x Faster and 45 MB Smaller

1. The Anatomy of the Problem: Why node_modules Kills Scanners

A typical developer machine might contain 20 to 100 repositories. Each repository often houses:

• node_modules (often containing 30,000–100,000 files per project)
• Build outputs (dist, .next, .turbo, build, target)
• Developer toolchains (.NET bin/obj, Xcode DerivedData, Docker build caches)

If you implement naive recursive traversal in Node.js, your scanner will inevitably choke:

// ❌ The Naive Approach: Explodes on 1,000,000 inodes
async function getDirSize(dir) {
  let size = 0;
  const entries = await fs.promises.readdir(dir, { withFileTypes: true });
  for (const entry of entries) {
    const fullPath = path.join(dir, entry.name);
    if (entry.isDirectory()) size += await getDirSize(fullPath);
    else size += (await fs.promises.stat(fullPath)).size;
  }
  return size;
}

Why this fails in production:

1. Node.js Event Loop Saturation: Thousands of fs.stat promises flood the libuv threadpool.
2. File Descriptor Starvation: Unbounded concurrency risks hitting OS EMFILE (too many open files) limits.
3. Massive Memory Pressure: Creating millions of V8 string paths and stat objects triggers aggressive Garbage Collection pauses.
4. Userland Inefficiency: Traversing millions of directory entries in userland JavaScript is orders of magnitude slower than kernel-space file lookups.

2. Scan Engine Redesign: From Minutes to Seconds

To fix this, we rearchitected the scan engine around three core principles: heuristic pruning, two-phase execution, and native kernel delegation.

A. Phase 1: Fast Heuristic Discovery (< 300ms)

Instead of sizing everything immediately, Phase 1 only searches for project roots. We crawl filesystem trees up to a maximum depth of 4, applying strict skip-lists:

const skipCrawlDirs = new Set([
  ...skipDirNames,
  "Library", "System", ".Trash", "Downloads", "Music", "Movies", "Pictures",
  ".npm", ".yarn", ".pnpm-store", ".cache", ".vscode", ".docker", ".cargo",
  "VirtualBox VMs", "Parallels"
]);

When the crawler hits a directory with a package.json, Cargo.toml, or .sln file, it registers candidate target folders (node_modules, dist, .next, bin, obj) without sizing them, and immediately stops recursing deeper into that project’s tree.

B. Phase 2: Native Kernel Offloading with du

Instead of recursing millions of nested files in Node.js, we offload directory block calculation to the operating system's native tools:

function getDirectorySizeFast(dirPath) {
  return new Promise((resolve) => {
    // 1. Fast path: 0ms mtime cache lookup
    try {
      const st = fsSync.statSync(dirPath);
      const cached = dirSizeCache.get(dirPath);
      if (cached && cached.mtimeMs === st.mtimeMs) {
        return resolve(cached.sizeBytes);
      }
    } catch {}

    // 2. Native POSIX kernel delegation on macOS/Linux
    if (process.platform === "darwin" || process.platform === "linux") {
      execFile("du", ["-sk", dirPath], { timeout: 6000 }, async (err, stdout) => {
        let sizeBytes = 0;
        if (!err && stdout) {
          const match = stdout.trim().match(/^(\d+)/);
          if (match) sizeBytes = parseInt(match[1], 10) * 1024;
        }
        if (sizeBytes === 0) sizeBytes = await getDirectorySizeBytes(dirPath, 20);
        
        dirSizeCache.set(dirPath, { mtimeMs: st.mtimeMs, sizeBytes });
        persistScanCache();
        resolve(sizeBytes);
      });
    }
  });
}

Because du -sk reads filesystem inodes directly in C at the kernel layer, measuring a 1.2 GB node_modules directory takes ~15ms instead of ~400ms.

C. Bounded Concurrency Worker Pool

Measuring 50 projects simultaneously with du could trigger CPU contention or spawn too many subprocesses. We built an asynchronous worker pool with a strict concurrency limit:

const CONCURRENCY_LIMIT = 8;
const workers = Array(CONCURRENCY_LIMIT).fill(0).map(async () => {
  while (nextTargetIndex < allTargets.length) {
    const item = allTargets[nextTargetIndex++];
    const sizeBytes = await getDirectorySizeFast(item.target.path);
    item.target.sizeBytes = sizeBytes;
    completedTargets++;
    // Real-time progress streamed to UI
    onProgress({ phase: "measuring", completedTargets, totalTargets });
  }
});
await Promise.all(workers);

3. Shrinking the Binary: Cutting ~45 MB of Bloat

Electron provides a complete Chromium and Node.js runtime. However, default packaging tools bundle dozens of megabytes of files your users will never touch.

A. Pruning Unused Chromium Locales (~45 MB Saved)

Chromium ships with over 50 localized .lproj directories (on macOS) and .pak files (on Windows) for international language support. If your application is in English, all 50+ language packs sit unused in the app bundle.

function stripUnusedLocales(appPath) {
  const keep = new Set(["en.lproj", "en_GB.lproj", "Base.lproj"]);
  const targets = [
    path.join(appPath, "Contents", "Resources"),
    path.join(appPath, "Contents", "Frameworks", "Electron Framework.framework", "Versions", "A", "Resources"),
  ];

  let removedCount = 0;
  for (const dir of targets) {
    if (!fs.existsSync(dir)) continue;
    for (const item of fs.readdirSync(dir)) {
      if (item.endsWith(".lproj") && !keep.has(item)) {
        fs.rmSync(path.join(dir, item), { recursive: true, force: true });
        removedCount++;
      }
    }
  }
  console.log(`🧹 Pruned ${removedCount} unused Chromium language packs (~45 MB saved)`);
}

B. Maximum-Compression DMG Image Generation (UDZO Level 9)

On macOS, standard DMG tools often use default compression levels. We tuned hdiutil to use UDZO (zlib compressed image) with maximum level 9 compression:

hdiutil create -srcfolder SpaceWise.app -ov -format UDZO -imagekey zlib-level=9 SpaceWise-arm64.dmg

This reduced download sizes by an additional 15–20%, ensuring that developers on mobile hotspots or restricted corporate networks can download SpaceWise in seconds.

4. Summary of Optimization Results

5. Key Lessons for Desktop App Developers