Crawlytics

Crawlytics is a high-performance, full-stack web scraping and content analytics platform. Built for speed and precision, it allows users to transform raw URLs into structured knowledge assets with AI-driven insights.

Features

• High-Fidelity Scraping: Powered by Firecrawl, Crawlytics extracts clean markdown and structured JSON from even the most complex web pages.

• Intelligent Mapping: Map entire website architectures and discover deep links with a single click.

• Streaming Overviews: Real-time AI Executive Summaries powered by Vercel AI SDK. Watch as the system synthesizes complex web content into concise insights instantly.

• Intelligent Knowledge Tagging: Automated content categorization. The system analyzes summaries to generate relevant, high-impact tags for your library.

• Collaborative Library: A unified, searchable command center for your entire collection of web data.

• Premium UX/UI: A futuristic, glassmorphism-inspired interface featuring:

• Zero-Scroll Dashboards: Fixed-view content readers for maximum focus.

• Sticky Metadata Sidebars: Keep asset context pinned while you read.

• Advanced Keyboard Shortcuts: Power-user features like Cmd+K for global search.

• Type-Safe Architecture: End-to-end type safety from the scraping engine to the dashboard UI.

Crawlytics uses a Server Functions architecture via TanStack Start, enabling seamless data fetching and scraping operations without the need for a separate API layer.

• src/data/: Server-side logic for Prisma operations and Firecrawl integration.

• src/routes/: File-based routing, metadata management, and AI streaming endpoints (/api/ai/summary).

• src/components/: Modular UI system with specialized AI-response components.