Description
Overview
This automation workflow demonstrates a dual AI agent orchestration pipeline focused on web scraping and API-driven activity suggestion. It targets developers and automation architects seeking no-code integration of AI-powered data extraction and dynamic content retrieval using HTTP-based tools and OpenAI language models.
The workflow is triggered manually and incorporates an HTTP Request node configured with API key authentication to scrape web pages and another to fetch filtered activity suggestions via REST API calls.
Key Benefits
- Enables dynamic web scraping through an AI agent using a REST API-based HTTP request tool.
- Integrates API-driven activity suggestions filtered by type and participant count in a no-code integration pipeline.
- Leverages OpenAI language models to interpret prompts and orchestrate tool calls automatically within the automation workflow.
- Reduces workflow node complexity by replacing subworkflows and manual response formatting with streamlined tool usage.
Product Overview
This automation workflow initiates via a manual trigger node, starting two parallel AI agents configured with explicitly defined prompts. The first AI agent receives a request to scrape the latest 10 issues from a GitHub repository, invoking an HTTP Request node to call the Firecrawl API. This API call is a POST request with a JSON body specifying the target URL and options to retrieve only the main content while removing media tags and converting relative paths to absolute ones. The response is returned in markdown format for structured analysis.
The second AI agent processes a prompt asking for activity suggestions, which it fulfills by querying the Bored API with parameters for activity type and participant number. This request uses query string parameters and fetches filtered results accordingly. Both agents use OpenAI’s chat models as their language model backend, receiving the input prompt and deciding when to call their respective HTTP request tools.
Error handling relies on n8n’s default node retry and failure management mechanisms, with no custom retry or backoff configured. API authentication is handled via HTTP header authentication credentials securely stored in the workflow environment. Data is processed transiently without persistence beyond the workflow’s execution cycle.
Features and Outcomes
Core Automation
This no-code integration pipeline accepts textual prompts as inputs and uses AI agents to determine appropriate API calls. The agents apply deterministic decision-making to invoke either a web scraping tool or an API activity filter based on prompt content.
- Single-pass evaluation of input prompts to select relevant HTTP request tool invocation.
- Explicit prompt definitions via ‘Set’ nodes ensure controlled input content for reproducible output.
- Parallel execution paths enable simultaneous data retrieval from multiple sources.
Integrations and Intake
The orchestration pipeline integrates with two external APIs using HTTP Request nodes authenticated via HTTP header credentials. Input event is a manual trigger, with prompt text set explicitly in the workflow.
- Firecrawl API for webpage scraping with POST JSON body containing URL and content filtering options.
- Bored API for retrieving filtered activity suggestions via query parameters.
- OpenAI API for chat-based language model processing secured via API key credential.
Outputs and Consumption
The workflow outputs structured text responses from AI agents enriched by API data. Responses are synchronous within the workflow execution and include markdown-formatted scraped content and JSON activity suggestion lists.
- Markdown content extracted from web pages with media tags removed for cleaner analysis.
- JSON arrays of activity suggestions filtered by type and participant number.
- AI-generated textual summaries or recommendations based on aggregated input data.
Workflow — End-to-End Execution
Step 1: Trigger
The workflow begins with a manual trigger node initiated by user interaction. This trigger simultaneously activates two ‘Set’ nodes that define distinct chat input strings for each AI agent.
Step 2: Processing
Each ‘Set’ node assigns a predefined prompt string representing distinct user queries. These prompts are passed unchanged to the respective AI agent nodes without additional schema validation, relying on deterministic input assignment.
Step 3: Analysis
AI agent nodes use the OpenAI chat model to process the input text and decide whether to invoke external tools. The web scraping agent calls the Firecrawl HTTP Request tool, passing URL and page options, while the activity suggestion agent calls the Bored API with query parameters.
Step 4: Delivery
Each AI agent receives the API response and formats the data internally. The workflow returns these processed results synchronously upon completion of both parallel branches, making structured data available for further consumption or downstream processing.
Use Cases
Scenario 1
A developer needs to retrieve the latest GitHub issues programmatically without manual web browsing. This workflow’s web scraping agent calls a scraping API to extract markdown content from the issues page, returning structured data in one workflow execution cycle.
Scenario 2
An automation architect wants to suggest suitable activities based on user preferences. The second AI agent dynamically queries an activity API filtered by type and participant count, delivering targeted suggestions through an event-driven analysis pipeline.
Scenario 3
Teams building AI agents require simplified workflows that reduce complexity and eliminate manual response formatting. This dual-agent orchestration pipeline replaces subworkflows with integrated tools, streamlining AI-driven data retrieval and processing.
How to use
To deploy this automation workflow, import it into an n8n environment with configured credentials for the OpenAI API, Firecrawl API, and Bored API. Trigger the workflow manually to initiate parallel AI agents. Customize the input prompts by editing the ‘Set ChatInput’ nodes to suit specific use cases. The workflow will execute synchronously, returning structured markdown content from web scraping and JSON-based activity suggestions. Monitor execution logs for error details and ensure API keys remain valid for uninterrupted operation.
Comparison — Manual Process vs. Automation Workflow
| Attribute | Manual/Alternative | This Workflow |
|---|---|---|
| Steps required | Multiple manual steps including browsing, data extraction, and API calls | Single manual trigger initiates automated AI agent orchestration |
| Consistency | Subject to human error and variable data formatting | Deterministic prompt-based AI decision-making with standardized API responses |
| Scalability | Limited by manual labor and browser automation complexity | Scales through parallel AI agents and programmatic API integration |
| Maintenance | Requires ongoing manual updates and error handling | Centralized credential management and simplified node structure reduce upkeep |
Technical Specifications
| Environment | n8n automation platform |
|---|---|
| Tools / APIs | OpenAI Chat Model, Firecrawl Webscraper API, Bored API |
| Execution Model | Manual trigger with synchronous parallel processing |
| Input Formats | Plain text prompts defined in workflow ‘Set’ nodes |
| Output Formats | Markdown content and JSON responses |
| Data Handling | Transient processing with no persistent storage |
| Known Constraints | Relies on external API availability and valid credentials |
| Credentials | HTTP header API keys for Firecrawl and OpenAI |
Implementation Requirements
- Valid API credentials for OpenAI, Firecrawl, and Bored API configured in n8n.
- Network access allowing outbound HTTPS requests to external APIs.
- Manual trigger activation to start the workflow and define prompt inputs.
Configuration & Validation
- Verify that API credentials are correctly set in n8n’s credential manager for HTTP header authentication.
- Test manual trigger to ensure that ‘Set ChatInput’ nodes assign expected prompt strings without errors.
- Confirm that AI agent nodes successfully call their respective HTTP Request tools and receive valid responses.
Data Provenance
- Trigger node: ‘When clicking ‘Test workflow’’ initiates manual execution.
- AI agents: ‘AI Agent’ and ‘AI Agent1’ process chatInput and orchestrate tool calls.
- Tools: ‘Webscraper Tool’ (Firecrawl API) and ‘Activity Tool’ (Bored API) fetch external data.
FAQ
How is the automation workflow triggered?
The workflow is triggered manually via a manual trigger node, which initiates both AI agents in parallel based on predefined prompt inputs.
Which tools or models does the orchestration pipeline use?
The workflow uses OpenAI chat models as the language model backend and integrates two HTTP Request tools calling the Firecrawl API for web scraping and the Bored API for activity suggestions.
What does the response look like for client consumption?
Responses include markdown-formatted web-scraped content and JSON arrays of filtered activity suggestions, returned synchronously within the workflow execution.
Is any data persisted by the workflow?
No data persistence occurs; all data is transiently processed during workflow execution without storage beyond runtime.
How are errors handled in this integration flow?
Error handling defaults to n8n’s native retry and failure mechanisms; no custom error handling or backoff strategies are configured.
Conclusion
This automation workflow provides a dependable no-code integration pipeline for AI-driven web scraping and API-based activity suggestions. Utilizing OpenAI language models and authenticated HTTP Request tools, it executes deterministic, prompt-driven orchestration with transient data handling. The workflow requires manual triggering and valid external API credentials, relying on API availability for full functionality. Its streamlined node structure reduces complexity compared to legacy subworkflow designs, supporting maintainable automation with consistent, structured outputs for diverse applications.
Reviews
There are no reviews yet.