AI Agents

Create AI-ready files from site crawls

23 nodes 236 122 Automatic trigger
Download

Workflow Description

Advanced automation that transforms crawled website data into AI-ready files through automatic classification, summarization, and content extraction. Processes large-scale web data for machine learning models using OpenAI integration and intelligent filtering.

How it works

  1. 1.Trigger workflow via form submission with crawled website data
  2. 2.Classify content automatically using text classifier and AI injection
  3. 3.Summarize important files and convert to standardized formats
  4. 4.Extract key data from documents and organize hierarchically
  5. 5.Apply filters to remove irrelevant or duplicate content
  6. 6.Generate final files optimized for AI model consumption

Use cases

  • Preparing large website content repositories for AI model training and fine-tuning
  • Extracting structured data from web pages for intelligent search systems
  • Creating documented knowledge bases from manual and automated web crawling

Requirements

  • Active OpenAI API key for classification and summarization services
  • Configured source web form to receive crawled site data

Service Value

Ideal as a smart automation service combining integrations and AI to produce ready-to-use results.

Apps Used

If OpenAI Form Trigger

Details

Trigger Automatic trigger
Nodes 23
Apps 3
Views 236
Downloads 122

How to Use

  1. 1.Click "Download Template"
  2. 2.Open your n8n dashboard
  3. 3.Go to Workflows > Import from File
  4. 4.Select downloaded file and configure credentials

Nodes Used (23)

/

Sticky Note

Sticky Note

#1

Sticky Note1

Sticky Note

#2

Sticky Note2

Sticky Note

#3

Set useful fields

Set

#4

Text Classifier

If

#5

OpenAI Chat Model

OpenAI

#6

No Operation, do nothing

No Op

#7

Sticky Note3

Sticky Note

#8

Filter URLs

Filter

#9

Sticky Note4

Sticky Note

#10

Sticky Note5

Sticky Note

#11

Sticky Note6

Sticky Note

#12

Sticky Note7

Sticky Note

#13

Sticky Note8

Sticky Note

#14

Sticky Note9

Sticky Note

#15

Summarize - Concatenate

Summarize

#16

Set Fields - llms.txt Content

Set

#17

upload file anywhere

No Op

#18

Set Field - llms.txt Row

Set

#19

Sticky Note10

Sticky Note

#20

Form - Screaming frog internal_html.csv upload

Form Trigger

#21

Extract data from Screaming Frog file

Extract From File

#22

Generate llms.txt file

Convert To File

#23