A look at facebook-gql-scraper: profiles, groups, Marketplace and comments without browser overhead or selector breakage, and what to verify before you build on it.
Getting structured data out of Facebook programmatically usually means one of two painful things. Either you drive a real browser in headless mode, logging in, scrolling, and waiting for content to render, or you fetch pages and pick values out of the markup with selectors. The first costs memory, CPU and startup time on every worker. The second breaks quietly whenever the front end ships a new class name or restructures a component.
A recent open source project takes a different route. Its author, posting as mhloo on Hacker News, describes it as a Facebook scraper that is lightweight, does not use a headless browser, and does not rely on fragile HTML parsing. The code is on GitHub at XD-MHLOO/facebook-gql-scraper, and the author invites people to try it and contribute.
That combination of claims targets exactly the two failure modes above. Whether it suits your workload depends on what you need from Facebook and how much breakage you can absorb, so the claims deserve a closer look.
What the scraper is built to extract
According to the author's announcement, the tool covers a broad range of Facebook surfaces: profiles, posts, comments, reactions, groups, Marketplace and Facebook's own search. That scope goes well past fetching a public page and reading its posts. Comments and reactions are where engagement analysis lives, groups hold community discussion, and Marketplace is a listings dataset in its own right.
Breadth like this matters because many scrapers do one surface well and leave you stitching together a second tool for the rest. A single library that understands profiles, groups and Marketplace means one authentication path, one data model and one place to fix things when Facebook changes. Before planning around any one surface, check the repository's README and issues to see which of these are maintained and tested, because coverage of this kind rarely stays uniform across every surface.
Why dropping the headless browser matters
A headless browser is the brute-force answer to scraping a JavaScript-heavy site: run the whole client, let it render, then read the result. It works, but every concurrent session carries a full browser process. Scaling to many accounts or many queries means scaling browser instances, with the memory footprint, crash recovery and container image size that comes with them.
A scraper that talks to the site without a browser avoids that overhead. It can run as an ordinary process making network requests, which is cheaper to host, faster to start and easier to run in constrained environments such as small containers, scheduled jobs or serverless functions. That is what 'lightweight' buys in practice: less infrastructure between your code and the data.
Why avoiding HTML parsing matters
HTML scraping fails in a particularly expensive way. When the markup changes, selectors often keep matching something, just not the right thing, so the pipeline keeps running and starts writing empty fields or wrong values. You find out days later from a downstream report rather than from a stack trace.
The author's point that the tool 'doesn't rely on fragile HTML parsing' addresses that directly. The 'gql' in the repository name suggests it works with GraphQL responses rather than rendered pages; read the code to confirm how it actually fetches data. If it does consume structured responses, the payoff is data that arrives already shaped as fields instead of text pulled out of nested divs. Structured responses can still change, but a missing key tends to fail loudly, which is the failure you want.
Evaluating it before you depend on it
The project was announced as a personal release with an open call for contributors. The Hacker News post drew 23 points and three comments, which is interest, not a track record. Treat it the way you would any young open source dependency and check the basics yourself:
- License: confirm the repository's license allows your intended use, especially for commercial work.
- Authentication: find out whether it needs a logged-in session or cookies, and what that means for the accounts you use.
- Maintenance: look at recent commits, open issues and how quickly breakages get fixed.
- Output format: inspect what each surface returns and whether the fields cover your use case.
- Rate behaviour: see whether it throttles requests or leaves pacing to you.
Pin a specific commit or release rather than tracking the main branch. With a scraper, an upstream change can alter output shape, and you want that to happen when you choose to upgrade, not overnight.
Where this approach does not fit
Skipping the browser trades one kind of fragility for another. Instead of depending on rendered markup, a request-based scraper depends on the internal requests the site's own client makes. Those are not a public API, carry no stability guarantee, and can change without notice. When they do, the fix has to come from someone who reverse-engineers the new shape, which for a small project may be one maintainer.
It is also a poor fit when you need what only a real browser provides: visual output such as screenshots, or flows that depend on full client-side behaviour. And no scraper, however efficient, settles whether you are allowed to collect the data. Facebook's terms, the privacy of the people whose profiles and comments you collect, and data-protection law in your jurisdiction all apply regardless of the tooling. Where an official API exposes the data you need, it is the more defensible choice even if it returns less. Where it does not, this project is a lighter starting point than a browser farm, provided you budget for the breakages.
This article draws on Open Source Alternatives to Reflector? and answers to it (1, 2), on the Stack Exchange network, licensed under CC BY-SA 4.0.
Share this article
Comments