Description
Overview
This agent with custom HTTP request automation workflow enables precise extraction and transformation of webpage content into manageable Markdown format. Designed as an orchestration pipeline, it processes manual chat inputs that specify URLs and methods, fetching and simplifying HTML body content for streamlined text output suitable for AI or user consumption.
Key Benefits
- Converts complex webpage HTML into clean Markdown for easier downstream processing.
- Supports method-based simplification to remove links and images, reducing content length.
- Handles HTTP request errors gracefully, returning explanatory messages as needed.
- Extracts only the body content of webpages, excluding scripts and multimedia elements.
Product Overview
This no-code integration workflow begins with a manual chat trigger receiving a query string containing a URL and a method parameter. The workflow parses these parameters into JSON, applying a configurable maximum content length limit defaulting to 70,000 characters. It executes an HTTP GET request to the provided URL, tolerating unauthorized certificates and preventing workflow failure on request errors by returning informative error messages. Upon successful fetch, the workflow extracts the HTML body content using regex and cleans it by removing script, style, iframe, video, audio, and other non-text elements. If the method parameter includes a simplification flag, the workflow replaces all hyperlinks and image sources with placeholders (“NOURL” and “NOIMG”) to reduce complexity. The cleaned HTML is then converted to Markdown format, preserving document structure while maintaining a compact size. Finally, the workflow validates the Markdown length against the maximum limit, returning an error message if the content exceeds this constraint. This deterministic, synchronous request–response workflow ensures reliable extraction of readable page content for further AI-driven text analysis or display.
Features and Outcomes
Core Automation
The automation workflow accepts manual chat input containing a query string and applies deterministic parsing to extract URL and method parameters. It branches based on error detection from the HTTP request node, ensuring controlled handling of failed fetches. The extracted HTML body content undergoes heuristic cleaning and optional simplification before conversion to Markdown.
- Single-pass evaluation of HTML body extraction and cleaning.
- Conditional branching to handle HTTP errors gracefully.
- Configurable maximum output length to prevent oversized responses.
Integrations and Intake
This orchestration pipeline integrates an HTTP request node configured for GET operations with tolerance for unauthorized SSL certificates. It relies on manualChatTrigger for intake, parsing query strings into structured JSON with keys such as url, method, and maxlimit. The workflow requires a valid URL parameter for proper operation.
- Manual chat input triggers the workflow with query parameters.
- HTTP Request node fetches webpage content with error tolerance.
- Parameter parsing node converts query strings to JSON for downstream use.
Outputs and Consumption
The output is a Markdown-formatted string representing the cleaned webpage body content, optionally simplified by removing links and images. The workflow operates synchronously, returning either the Markdown content or a specific error message if content length exceeds the configured maximum. Key output fields include page_content and page_length.
- Markdown format output preserves readable document structure.
- Output includes page length for validation and diagnostics.
- Returns error string if content exceeds maximum allowed length.
Workflow — End-to-End Execution
Step 1: Trigger
The workflow initiates upon receiving a new manual chat message, which must include a query string with parameters such as url and method. This manual trigger node enables on-demand execution from user input or external systems.
Step 2: Processing
The query string is parsed into a JSON object extracting keys for URL, method, and an optional maxlimit. The configuration node sets default values, including a 70,000-character maximum page length if unspecified. Basic validation ensures the presence of a valid URL parameter before proceeding.
Step 3: Analysis
An HTTP GET request is issued to the encoded URL with options to allow unauthorized certificates and prevent error propagation. If an error property is detected in the response, the workflow constructs a stringified error message or a usage instruction. Otherwise, it extracts the HTML content inside the body tag using regex and removes extraneous tags such as script, style, and multimedia elements. If the method parameter indicates simplification, it replaces all hyperlinks and image sources with placeholders to reduce content complexity.
Step 4: Delivery
The cleaned HTML content is converted into Markdown format, retaining textual structure but minimizing size. The workflow then compares the Markdown length against the maximum limit, returning the Markdown content if within bounds or an error message if exceeded. Output fields sent downstream include page_content and page_length.
Use Cases
Scenario 1
When a user needs to analyze webpage content without visual clutter, this event-driven analysis workflow fetches only the body text, strips scripts and multimedia, and returns clean Markdown. This enables efficient AI summarization or indexing without manual preprocessing.
Scenario 2
For applications requiring simplified content without external links or images, the orchestration pipeline’s method parameter triggers a removal of URLs and image sources. This reduces token usage and focuses analysis on textual data, producing deterministic, streamlined outputs.
Scenario 3
In situations with unreliable URLs or network issues, the automation workflow detects HTTP errors and returns structured error messages or usage instructions. This prevents workflow failures and informs users or agents to adjust inputs accordingly.
Comparison — Manual Process vs. Automation Workflow
| Attribute | Manual/Alternative | This Workflow |
|---|---|---|
| Steps required | Multiple manual steps: fetch, extract, clean, convert. | Automated single pipeline from input to Markdown output. |
| Consistency | Variable, depends on manual handling and tools. | Deterministic extraction and cleaning with defined rules. |
| Scalability | Limited by manual throughput and error handling. | Scalable via automated, event-driven processing and error control. |
| Maintenance | Ongoing manual updates and script adjustments required. | Centralized workflow with configurable parameters and error nodes. |
Technical Specifications
| Environment | n8n workflow automation platform |
|---|---|
| Tools / APIs | Manual Chat Trigger, HTTP Request, Regex extraction, Markdown conversion |
| Execution Model | Synchronous request–response with error continuation |
| Input Formats | Manual chat message containing URL and method in query string format |
| Output Formats | Markdown text with page_content and page_length fields |
| Data Handling | Transient processing; no persistence of fetched content |
| Known Constraints | Requires valid URL parameter; max page content length enforced |
| Credentials | OpenAI API key for language model interaction |
Implementation Requirements
- Valid manual chat input with query string including url and method parameters.
- Configured OpenAI API credentials for language model nodes.
- Network access permitting HTTP requests including to URLs with unauthorized SSL certificates.
Configuration & Validation
- Verify manual chat input correctly provides query string parameters, especially a valid URL.
- Confirm HTTP Request node executes without unhandled failures and properly detects errors.
- Test different method parameter values to validate simplification and full content modes.
Data Provenance
- Trigger node: On new manual Chat Message (@n8n/n8n-nodes-langchain.manualChatTrigger)
- HTTP Request node performs URL fetch with error detection and allowUnauthorizedCerts enabled
- Output fields page_content and page_length originate from Markdown conversion and length check nodes
FAQ
How is the agent with custom HTTP request automation workflow triggered?
The workflow is triggered manually via a chat message containing a query string with parameters such as url and method. This enables on-demand event-driven analysis initiated by user input.
Which tools or models does the orchestration pipeline use?
The pipeline integrates an HTTP Request node for webpage fetching, a regex-based HTML body extractor, and an OpenAI language model node configured with the GPT-4 preview. These work together to process input and generate cleaned text output.
What does the response look like for client consumption?
The workflow returns Markdown-formatted text of the cleaned webpage body content under the page_content field, along with page_length indicating the content size. If the content exceeds limits, a specific error message is returned instead.
Is any data persisted by the workflow?
No data persistence occurs within the workflow. All webpage content is processed transiently during execution without storage to ensure data privacy and reduce footprint.
How are errors handled in this integration flow?
Errors from the HTTP Request node do not fail the workflow; instead, a conditional node detects errors and returns stringified error messages or usage instructions. This enables controlled error propagation and user feedback.
Conclusion
This agent with custom HTTP request automation workflow provides a deterministic, event-driven analysis solution for extracting, cleaning, and simplifying webpage content into Markdown format. It ensures reliable operation with error detection and configurable output length constraints, delivering consistent, structured text suitable for AI consumption or user review. However, the workflow depends on the availability and accessibility of external URLs and the correctness of input query parameters to function as intended.
Reviews
There are no reviews yet.