Description
Overview
The HN Who is Hiring Scrape workflow automates extraction and structuring of job listings from Hacker News posts, leveraging an event-driven analysis pipeline. Designed for developers, recruiters, and data engineers, this automation workflow addresses the challenge of manually parsing unstructured job postings by delivering a unified data structure. It initiates via a manual trigger node and employs the Algolia search API to precisely query “Ask HN: Who is hiring?” posts.
Key Benefits
- Automates retrieval of job listings using precise full-text search queries on Hacker News data.
- Transforms unstructured text data into structured JSON using a no-code integration with GPT-based parsing.
- Filters postings to process only recent job listings created within the last 30 days for relevancy.
- Supports flexible output delivery by writing structured job data directly to Airtable for downstream use.
Product Overview
This automation workflow is manually triggered and begins by sending a POST request to the Algolia search API, querying for the exact phrase “Ask HN: Who is hiring?” filtered to story-type posts with pagination and typo tolerance enabled. The JSON response’s “hits” array is split to process each post individually. Key metadata fields — including title, creation date, update date, and story ID — are extracted for filtering. Posts older than 30 days are excluded using a date comparison filter node to ensure data freshness.
For each relevant post, the workflow calls the official Hacker News API to retrieve detailed post content and associated comment IDs, which represent individual job listings. These child comments are fetched individually via HTTP requests, extracting raw text content. The workflow then sanitizes text by removing HTML entities, tags, and normalizing line breaks, preparing the data for downstream processing.
Using an OpenAI GPT model node configured for structured output, the cleaned text undergoes semantic parsing into a defined JSON schema capturing fields such as company name, job title, location, employment type, salary, description, and application URLs. The structured data is validated by a schema-based output parser node. Finally, the workflow asynchronously writes the parsed job listings to an Airtable base, enabling efficient data storage and retrieval. Error handling follows n8n’s default retry and failure mechanisms without additional customizations.
Features and Outcomes
Core Automation
This no-code integration pipeline ingests job post data from Hacker News via Algolia API, applying filters and text cleaning before transforming content into a structured JSON format using an AI language model.
- Single-pass evaluation of recent posts filtered by creation date within the workflow.
- Deterministic extraction of relevant fields including title, story ID, and timestamps.
- Automated branching to fetch individual job posts based on comment IDs.
Integrations and Intake
The orchestration pipeline integrates the Algolia search API with HTTP header authentication for querying Hacker News posts, and the official Hacker News REST API for fetching detailed story and comment data. Authentication for Algolia uses an HTTP header credential type, while Hacker News API calls require no authentication.
- Algolia API for targeted search queries and pagination of Hacker News posts.
- Hacker News API for retrieving detailed JSON of posts and their comments.
- OpenAI API integration using an API key credential for semantic structuring.
Outputs and Consumption
The workflow produces structured JSON output conforming to a defined schema, then asynchronously pushes this data to an Airtable base. Output fields include company, title, location, employment type, salary, description, and URLs. This enables downstream consumption for data analysis or job board population.
- Structured JSON data output validated against a manual schema.
- Asynchronous writing of job data to Airtable for persistent storage.
- Consistent field mappings for applications, company info, and job details.
Workflow — End-to-End Execution
Step 1: Trigger
The entire workflow is manually initiated via the “When clicking ‘Test workflow’” manual trigger node, allowing controlled execution on demand.
Step 2: Processing
After triggering, a POST request is sent to the Algolia search API with a JSON body specifying the exact phrase query and filters. The JSON response is split on the “hits” array, and relevant fields such as title and timestamps are extracted. Posts are filtered to include only those created within the last 30 days, ensuring recent job listing data is processed.
Step 3: Analysis
The workflow retrieves full post details and associated job comments from the Hacker News API. Text content from job listings is cleaned using JavaScript code to remove HTML entities and tags, normalize whitespace, and format URLs. This cleaned text is then fed into an OpenAI GPT model configured to extract structured job information in JSON format according to a predefined schema.
Step 4: Delivery
Structured job data parsed by the output parser node is delivered asynchronously to an Airtable base using API credentials. This final data write operation stores job listings with consistent fields for subsequent querying or integration.
Use Cases
Scenario 1
A recruiter needs to gather current tech job openings from Hacker News efficiently. This automation workflow queries and extracts recent “Who is hiring” posts, structures the data, and stores it in Airtable, enabling streamlined candidate sourcing.
Scenario 2
A developer wants to monitor job market trends by analyzing structured job listings from Hacker News. The orchestration pipeline converts raw posts into normalized JSON, supporting data-driven insights on hiring patterns.
Scenario 3
An HR analyst needs to automate job board updates with verified and recent Hacker News postings. This event-driven analysis flow filters for freshness, extracts job details, and outputs clean data ready for integration into internal systems.
How to use
To deploy this workflow within n8n, first configure the Algolia HTTP header authentication credentials with your Algolia API key. Set up OpenAI API credentials with a valid key for the GPT model node. Provide Airtable API access tokens and specify the target base and table for storing job data. Trigger the workflow manually to start scraping recent “Who is hiring” posts from Hacker News. The results will be cleaned, structured, and pushed to Airtable, ready for review or integration.
Comparison — Manual Process vs. Automation Workflow
| Attribute | Manual/Alternative | This Workflow |
|---|---|---|
| Steps required | Multiple manual API queries and manual text parsing for each job post. | Single automated pipeline from search query to structured data storage. |
| Consistency | Inconsistent formatting and human error in data transcription. | Deterministic text cleaning and structured schema validation. |
| Scalability | Limited by manual effort and time to process large volumes. | Scales to hundreds of posts with automated splitting and API calls. |
| Maintenance | High due to manual interventions and error corrections. | Low, leveraging reusable n8n nodes and stable API endpoints. |
Technical Specifications
| Environment | n8n workflow automation platform |
|---|---|
| Tools / APIs | Algolia Search API, Hacker News REST API, OpenAI GPT API, Airtable API |
| Execution Model | Manual trigger with asynchronous HTTP requests and API calls |
| Input Formats | JSON responses from search and REST APIs |
| Output Formats | Structured JSON data matching a defined schema |
| Data Handling | Transient processing with no persistent storage except Airtable output |
| Known Constraints | Relies on availability of external APIs (Algolia, Hacker News, OpenAI, Airtable) |
| Credentials | HTTP header authentication for Algolia, API keys for OpenAI and Airtable |
Implementation Requirements
- Valid Algolia API key configured with HTTP header authentication in n8n.
- OpenAI API key with access to GPT models for text structuring.
- Airtable personal access token with write permissions to target base and table.
Configuration & Validation
- Confirm Algolia credentials are correctly set and test the search query for expected results.
- Verify Hacker News API calls return valid JSON for story and comment IDs.
- Test OpenAI node output parser with sample text to ensure structured JSON matches schema.
Data Provenance
- Manual trigger node initiates the workflow execution.
- “Search for Who is hiring posts” node queries Algolia API using HTTP header authentication.
- OpenAI Chat Model node processes cleaned text to produce structured JSON adhering to a manual schema.
FAQ
How is the HN Who is Hiring Scrape automation workflow triggered?
The workflow is triggered manually via the n8n manual trigger node, allowing users to run the process on demand.
Which tools or models does the orchestration pipeline use?
The workflow integrates the Algolia Search API for querying posts, the Hacker News API for detailed data retrieval, and OpenAI’s GPT model for converting unstructured text into structured JSON data.
What does the response look like for client consumption?
The output is structured JSON containing fields such as company, job title, location, employment type, salary, description, and application URLs, stored asynchronously in Airtable.
Is any data persisted by the workflow?
Data is transiently processed within the workflow. Persistent storage occurs only in the Airtable base where structured job listings are saved.
How are errors handled in this integration flow?
Error handling relies on n8n platform defaults, including automatic retries and failure notifications; no custom error handling nodes are configured.
Conclusion
The HN Who is Hiring Scrape workflow automates the end-to-end extraction and structuring of job listings from Hacker News “Ask HN: Who is hiring?” posts, providing deterministic and consistent output in JSON format. Its manual trigger ensures controlled execution, while integration with Algolia, Hacker News API, OpenAI GPT, and Airtable enables a seamless data transformation pipeline. The workflow depends on the continuous availability of external APIs and requires proper credential setup for full functionality. This solution reduces manual effort, enforces data consistency, and supports scalable job data aggregation with minimal maintenance.
Reviews
There are no reviews yet.