Layer 3 · SearXNG

Every enabled engine, in parallel, from your IP address.

SearXNG turns a query into candidate URLs by asking a lot of search engines at once and merging what comes back. There is no sequential fallback. That single design fact explains both why this layer is cheap and why it fails the way it does.

This is the layer that will break. Not because SearXNG is fragile, but because scraping search engines from a residential IP under sustained agent traffic is a thing search engines actively defend against. The repository's response is not to pretend otherwise — the README has a section titled When search returns nothing that opens with "This will happen."

TopologyThe fan-out

SearXNG engine fan-out: three bands of engine, all queried in parallel crw sends one query to searxng, which fans out simultaneously to the default general engine set, the six independent crawlers enabled by this repository, and any API-backed engines that have been uncommented. Results merge and return on a single path. The default set is marked as the group that suspends together. crw one query searxng queries every enabled engine in parallel request_timeout 3.0 retries 3 · limiter off DEFAULT GENERAL SET — SCRAPED, FROM YOUR IP google brave duckduckgo startpage ✕ CAPTCHA and rate-limit together under agent traffic …plus the rest of a registry of roughly 250 engines, most of them category-specific rather than general web. INDEPENDENT CRAWLERS — TURNED ON BY THIS REPO mojeek marginalia stract mwmbl right dao yep ✓ own indexes — they do not share the majors' rate limits API-BACKED — SHIPPED COMMENTED OUT braveapi — engine: braveapi · $5 / 1k queries google api — engine: google_cse · needs api_key + cx serper — engine: json_engine · POST + header auth · $1 / 1k merged and ranked · failures reported in unresponsive_engines[] ONE QUERY IN · NO SEQUENTIAL FALLBACK · EVERY ENABLED ENGINE BILLS
Figure 5 — the fan-out The parallelism is the whole design, and it has a billing consequence worth stating plainly: because there is no sequential fallback, a second paid API key is not a backup. It is a second charge on every single search.

Failure modeWhen search returns nothing

The dangerous property of this failure is not that it happens, but that it is quiet. Individual engines being suspended is normal and invisible — the merge still returns results. When the default general engines suspend together, the merge returns an empty list, and every layer above it faithfully reports zero results with no error anywhere.

diagnosisREADME.md · scripts/health.sh lines 7–18
# the one-liner from the README
$ curl -s "http://127.0.0.1:8888/search?q=test&format=json" | jq .unresponsive_engines

# or let the health script say it — it checks this layer first, alone
$ ./scripts/health.sh
── searxng ──
   FAIL — 0 results. Engines are probably CAPTCHA'd/rate-limited:
      ['google', 'CAPTCHA']
      ['brave', 'HTTP error']
      ['duckduckgo', 'CAPTCHA']       # exact shape varies by build
     See README → 'When search returns nothing'.
── crw ──
   OK — v(your pinned image)

Note what that transcript shows: crw is fine. The engine is healthy, the container is up, scraping works. Only the layer that finds URLs is broken. This is exactly why health.sh checks four things separately instead of running one end-to-end query — a single composite check would have told you "search is broken" and left you reading crw logs.

The fixes, ranked by how much they actually help

Ordering and wording follow README.md, "When search returns nothing". Only the first one changes the failure from frequent to rare.
fixwhat it doeshonest verdict
1. Add API-backed engines Engines that authenticate instead of scraping keep working when the scraped ones are suspended The only fix that makes search reliable
2. Keep the extra crawlers on mojeek, marginalia, stract, mwmbl, right dao, yep — independent indexes, different limits The difference between fewer results and zero
3. Set useragent_suffix Puts a real contact address in your User-Agent SearXNG's docs: it helps "if an engine wants to block you"
4. Wait Nothing Suspensions are temporary
5. Route through a different IP ssh -D 1080 user@host is a free SOCKS5 proxy on a machine you already own; SearXNG rotates round-robin across outgoing.proxies and picks a different one on each retry Works, if you have somewhere to route through
Do not reach for a free proxy list

Operators log traffic and inject content, and cheap "residential" pools are frequently sourced from malware-infected devices rather than opt-in consent. You would be routing your searches through someone else's compromised router — in a setup where the same agent may be holding your credentials. The proxy fix is only a fix if you own the exit.

ConfigurationThree ways to fail silently

Adding an API-backed engine to SearXNG is four lines of YAML and about an hour of your life, because all three of the ways it goes wrong produce no error message. The repository documents them in the config file itself, next to the block you are about to uncomment.

Gotcha 1 — inactive: false is required

Upstream ships API engines with inactive: true, which removes them from the engine registry entirely. It is stronger than disabled and is not overridden by disabled: false. Without it the engine simply never exists — no error, no log line, and even its !bang shortcut is unrecognised.

Gotcha 2 — !!env VAR does not work

Current SearXNG builds hard-fail settings load with could not determine a constructor for the tag tag:yaml.org,2002:env and the container restart-loops. Inline the key in the file — it is gitignored in a real deployment — or template it in before the container starts. This one at least fails loudly, but it fails as a crash loop rather than a config error.

Gotcha 3 — braces in request_body must be doubled

json_engine runs the body through Python's .format(), so a literal { opens a placeholder and you get KeyError: '"q"' at json_engine.py: request_body.format(**fp). Write {{ and }} for literal JSON braces. Nothing upstream documents this.

Why request_body braces must be doubled in json_engine Two pipelines. In the first, a single-braced request body reaches Python's str.format, which reads the opening brace as a placeholder and raises KeyError. In the second, doubled braces survive format as literal braces and produce valid JSON for the POST. SETTINGS.YML VALUE → PYTHON str.format(**fp) → WHAT THE ENGINE POSTS ✕ single braces request_body: '{"q": "{query}", "num": 10}' json_engine.py request_body.format(**fp) KeyError: '"q"' the leading { opened a placeholder Result: the engine contributes nothing. No results row, no obvious log line — you are left staring at a search that returns fewer results than it did yesterday. ✓ doubled braces request_body: '{{"q": "{query}", "num": 10}}' json_engine.py request_body.format(**fp) {"q": "rust async traits", "num": 10} POST https://google.serper.dev/search X-API-KEY: … · results_query: organic {{ and }} survive .format() as literal braces; {query} is the one real placeholder.
Figure 6 — the doubled-brace pipeline This is the general pattern for adding any JSON search API to SearXNG without writing an engine module: json_engine supports method, headers and request_body, which between them cover POST APIs with header authentication.
config/searxng/settings.ymllines 113–129, uncommented
113 - name: serper
114   engine: json_engine
115   shortcut: srp
116   categories: [general, web]
117   inactive: false
118   disabled: false
119   timeout: 5.0
120   search_url: https://google.serper.dev/search
121   method: POST
122   headers:
123     X-API-KEY: 'YOUR-SERPER-KEY'
124     Content-Type: application/json
125   request_body: '{{"q": "{query}", "num": 10}}'
126   results_query: organic
127   url_query: link
128   title_query: title
129   content_query: snippet
lines 126–129 — the shape adapter
Four jq-ish paths are all it takes to map a foreign JSON response onto SearXNG's result model: where the list lives, and which keys hold URL, title and snippet. No Python.
line 119 — timeout: 5.0
Higher than outgoing.request_timeout of 3.0 at line 32, because a paid API you are waiting on is worth a longer leash than a scraped page you can afford to lose.
line 125 — num: 10
Not arbitrary. Serper bills 1 credit for up to 10 results and 2 credits for 11–100, so asking for 11 doubles the price of every search for one extra link.
Pricing as recorded in config/searxng/settings.yml lines 104–107 and README.md. Both providers change prices; check before you commit to one.
optionengine:costnotes
Serperjson_engine$1 / 1k2,500/mo free tier, no card required; POST with header auth
Brave Search APIbraveapi$5 / 1kNative engine module, results_per_page: 20
Google CSEgoogle_cse—Needs both api_key and a cx engine ID

ConfigurationThe rest of the file

Everything not about engines is about being a private instance that a machine talks to, rather than a public one that people use.

config/searxng/settings.ymllines 9–44, abridged
13  # JSON must be enabled explicitly — SearXNG returns 403 on ?format=json
14  # otherwise, and every downstream call silently fails.
16  formats:
17    - html
18    - json
…
22  bind_address: "0.0.0.0"   # container-internal; compose publishes to loopback
26  limiter: false
27  public_instance: false
28  image_proxy: false
…
32  request_timeout: 3.0
36  useragent_suffix: ""
37  retries: 3
lines 16–18 — json
The single most important line in the file. Without it, ?format=json returns 403, crw gets nothing, and nothing in the stack says why. html stays enabled so the web UI at 127.0.0.1:8888 remains usable for debugging by hand.
line 22 — 0.0.0.0
Looks alarming, is not: this is the bind inside the container. The compose file publishes it as 127.0.0.1:8888, so the only interface it is reachable on is your loopback.
line 26 — limiter: false
SearXNG's own rate limiter needs Valkey or Redis. Skipped deliberately — one less container for a private instance. The file's comment is emphatic: do not skip it if you expose the instance.
line 36 — useragent_suffix
Empty by default and worth filling in with a real contact address. It makes you look like an identifiable small instance rather than an anonymous scraper.

SearXNG queries every enabled engine in parallel — there is no sequential fallback, so a second paid key is not a "backup", it is a second bill on every search.

README.md — When search returns nothing