← All projects
NCMA / WEB RESEARCH / SEARXNG + CRAWL4AI
Research with sources attached.
Self-hosted search and extraction feed a source-aware research path, with citations retained through synthesis and private infrastructure kept out of the public interface.
MY CONTRIBUTION
Designed and integrated the search, extraction and source-preservation path for research within the PersonalAgent context.
TOOLS & METHODS
SearXNG · Crawl4AI · self-hosted infrastructure · source collection · synthesis with citations
CHALLENGE
A research answer needs inspectable sources and an extraction path that can be checked independently of synthesis. Infrastructure access and private operational details should not become public evidence assets.
DESIGN DECISIONS
Separate search, extraction, source collection and synthesis. Preserve citations and carry source context forward instead of accepting an unsupported summary.
CONSTRAINTS
Research needs source provenance and controlled infrastructure. Extraction quality varies by page; source collection, synthesis and citations must remain independently inspectable.
The workflow, step by step.
01 / Search
Use SearXNG to collect candidate sources.
02 / Extract
Use Crawl4AI to obtain relevant page content.
03 / Retain sources
Carry origin and source context into the research evidence.
04 / Synthesize + cite
Build a grounded response with citations and explicit limits.
IMPLEMENTATION & VALIDATION
Self-hosted SearXNG finds sources and Crawl4AI extracts content. The workflow retains source context for grounded synthesis and citations. Recorded infrastructure and extraction evidence are separate from the pending physical synthesis smoke test.
RESULT & EVIDENCE
The recorded implementation state reports infrastructure and extraction validation. Physical end-to-end synthesis smoke remains pending and is not presented as a completed product acceptance.
EVIDENCE LIMIT
This Figma case is a conceptual workflow. It does not call a search service or claim live synthesis, current private-service availability or final physical acceptance.
VALIDATION BOUNDARY
Search and extraction checks are distinct from physical end-to-end acceptance. The case keeps that distinction visible.
PUBLIC VIEW / PRIVATE INFRASTRUCTURE
Only the logical workflow is public. Endpoints, authorization details, payloads and private host information are excluded.
TRADE-OFFS & LESSONS
Self-hosting gives infrastructure control and creates maintenance work. Extraction success does not prove synthesis quality. The public case exposes the logical flow while keeping private endpoints and source payloads out.
Explore architecture ↗
Next case: Control Room ↗
About the work and career ↗
VALTER LOURENÇO JUNIOR / NCMA SYSTEMS
← All projects
NCMA / WEB RESEARCH / SEARXNG + CRAWL4AI
Research with sources attached.
Self-hosted search and extraction feed a source-aware research path, with citations retained through synthesis and private infrastructure kept out of the public interface.
MY CONTRIBUTION
Designed and integrated the search, extraction and source-preservation path for research within the PersonalAgent context.
TOOLS & METHODS
SearXNG · Crawl4AI · self-hosted infrastructure · source collection · synthesis with citations
CHALLENGE
A research answer needs inspectable sources and an extraction path that can be checked independently of synthesis. Infrastructure access and private operational details should not become public evidence assets.
DESIGN DECISIONS
Separate search, extraction, source collection and synthesis. Preserve citations and carry source context forward instead of accepting an unsupported summary.
CONSTRAINTS
Research needs source provenance and controlled infrastructure. Extraction quality varies by page; source collection, synthesis and citations must remain independently inspectable.
The workflow, step by step.
01 / Search
Use SearXNG to collect candidate sources.
02 / Extract
Use Crawl4AI to obtain relevant page content.
03 / Retain sources
Carry origin and source context into the research evidence.
04 / Synthesize + cite
Build a grounded response with citations and explicit limits.
IMPLEMENTATION & VALIDATION
Self-hosted SearXNG finds sources and Crawl4AI extracts content. The workflow retains source context for grounded synthesis and citations. Recorded infrastructure and extraction evidence are separate from the pending physical synthesis smoke test.
RESULT & EVIDENCE
The recorded implementation state reports infrastructure and extraction validation. Physical end-to-end synthesis smoke remains pending and is not presented as a completed product acceptance.
EVIDENCE LIMIT
This Figma case is a conceptual workflow. It does not call a search service or claim live synthesis, current private-service availability or final physical acceptance.
VALIDATION BOUNDARY
Search and extraction checks are distinct from physical end-to-end acceptance. The case keeps that distinction visible.
PUBLIC VIEW / PRIVATE INFRASTRUCTURE
Only the logical workflow is public. Endpoints, authorization details, payloads and private host information are excluded.
TRADE-OFFS & LESSONS
Self-hosting gives infrastructure control and creates maintenance work. Extraction success does not prove synthesis quality. The public case exposes the logical flow while keeping private endpoints and source payloads out.
Explore architecture ↗
Next case: Control Room ↗
About the work and career ↗
VALTER LOURENÇO JUNIOR / NCMA SYSTEMS