Markdown for Agents and Statistics

سوالات متداول

Where are the Markdown files stored?

Inside wp-content/uploads/{export_dir}/ (configurable in Settings). Post files
live under {export_dir}/{post-type}/{slug}.md. Taxonomy archive files live under
{export_dir}/taxonomy/{taxonomy}/{term-slug}.md. Negotiated page requests are
served through WordPress. Direct uploads URLs are normally served by the web
server and bypass the plugin headers and counters.

Will this slow down my site?

No. Markdown files are generated ahead of time (on post save or via manual/CLI
bulk generation). Serving them is a simple file read, much faster than rendering
a full WordPress page.

Does bulk generation need anything special on my host?

Bulk generation (the “Generate everything” and per-post-type/taxonomy buttons)
runs in the background via WP-Cron, so WP-Cron needs to be working on the site.
If you have set DISABLE_WP_CRON and use a system cron to call wp-cron.php
instead, that cron must run as the same user as your web server, or the files
it writes will fail permission checks.

If WP-Cron is not working, a run sits at “running” with little or no progress
instead of failing outright — there is no page load to trigger the next batch.
Two things to check: whether DISABLE_WP_CRON is set with no system cron
actually running behind it, and whether a firewall or HTTP auth on the site is
blocking the loopback request WordPress makes to its own wp-cron.php.

How do I stop a bulk generation run that’s in progress?

There is no cancel button, but deactivating the plugin (Plugins screen, no CLI
needed) stops a run immediately: it clears the job and unschedules its
background events. Reactivating leaves already-generated files untouched, and
you can start a new run from Settings. Note this is blunt — deactivating also
stops the plugin serving Markdown to agents until you reactivate it.

AI agents are getting HTML instead of Markdown. Why?

First confirm the post is public, eligible, and has a current export. Missing
files or excluded posts legitimately fall back to HTML. Otherwise inspect each
CDN, firewall and page cache: it may answer before WordPress runs, return cached
HTML or block the request. A UA-only request can receive warm cached HTML even
when the HTML response varies on Accept.

The advertised ?output_format=md alternate is the preferred route when every
cache preserves that query parameter or explicitly bypasses it. It is not a
firewall bypass, and caches configured to ignore query strings can still answer
with the HTML entry. Direct upload URLs have separate static-cache behaviour.

How do I let my CDN or cache serve Markdown to agents?

Configure every cache layer, including any hosting-provider cache behind a CDN.
Bypass requests containing a Markdown Accept header, the output_format query
parameter, or a configured AI User-Agent where that layer supports the match.
Do not add User-Agent to the cache key: that fragments caching for every visitor.
Keep origin cache-control respected, and permit intended agents through security
controls separately from cache rules.

For LiteSpeed Cache, under Cache → Excludes, add the export directory (default
/wp-content/uploads/wp-mfa-exports/) to Do Not Cache URIs, output_format to
Do Not Cache Query Strings, and all configured agent substrings to Do Not Cache
User Agents, one per line. The Accept route additionally needs the README’s
rewrite rule. Regenerate UA exclusions whenever the configured list changes.

The CDN and cache guide
includes the worked configuration, other cache vendors, static-export caveats
and both-order verification. Negotiated Markdown’s private/no-store headers
prevent storage only when the cache honours them; a forced TTL can defeat them.

What is different about Cloudflare?

If using cache-everything, choose an Edge TTL that respects origin cache-control.
Put a Bypass cache rule for the Markdown query or configured UA after it:
for conflicting cache settings the last matching rule wins. Generate the UA
expression from this installation’s saved options, not a stale copied list.
Review bot/security policies and which layer manages robots.txt independently.

A controlled free-plan test on 21 September 2026 rejected an Accept-header Cache
Rule. Listed UA and query bypass worked; an unlisted client asking only through
Accept still received warm cached HTML. This was a related Wagtail deployment,
not verification of your WordPress host. Check current plan capabilities and
use the guide’s procedure against your actual stack.

How can I check what an agent actually receives?

Use GET requests from an external network against a public post with an export:

`

H=https://example.com/your-post/

Browser twice, to observe whether HTML really becomes cached.

curl -sS -D – -o /dev/null -A ‘Mozilla/5.0’ -H ‘Accept: text/html’ “$H”
curl -sS -D – -o /dev/null -A ‘Mozilla/5.0’ -H ‘Accept: text/html’ “$H”

Listed UA-only must receive Markdown even after cached HTML.

curl -sS -D – -o /dev/null -A ‘GPTBot/1.4’ -H ‘Accept: text/html’ “$H”
curl -sS -D – -o /dev/null “$H?output_format=md”

Unlisted Accept-only can receive HTML on Cloudflare free.

curl -sS -D – -o /dev/null -A ‘MarkdownVerification/1.0’ -H ‘Accept: text/markdown’ “$H”

Browser after agents must still receive HTML.

curl -sS -D – -o /dev/null -A ‘Mozilla/5.0’ -H ‘Accept: text/html’ “$H”
`

Check status, Content-Type, cache headers and cache-hit indicators. Negotiated
plugin responses carry X-Markdown-Source: markdown-for-agents; direct static
exports need not. HTML alone does not prove a cache fault: check eligibility and
origin evidence too. Stop testing if a browser receives Markdown.

Also test agent first, then browser, using a confirmed uncached public URL or a
specifically agreed purge. Repeat for each trigger; ordering alone does not prove
cold cache state. The guide covers direct exports, HEAD checks and correlation.

What do the access statistics count?

The plugin counts singular-post Markdown GET selections made by WordPress, not
all agent traffic or confirmed body delivery. HEAD and other methods, taxonomy
archives, HTML fallback, direct static exports and bundles do not increment
these counters. Neither do cache hits or requests blocked before WordPress.
Versions up to 1.7.2 also counted negotiated singular HEAD requests; the GET-only
correction is listed under Unreleased below.

Complete discovery and probes before a baseline snapshot because GET probes can
increment counters. For reconciliation, verify both correlation IDs in actual
origin records and compare UTC counter buckets after traffic has finished.
Document background traffic and CDN/static bypasses separately. Sending custom
headers does not make them appear in the usual combined access log.

Should I publish an llms.txt file?

llms.txt is a proposed convention for a single Markdown index of your site at
https://example.com/llms.txt, aimed at AI tools that look for a site-level
manifest. It is an emerging community convention, not an official standard, and
there is limited evidence that the major AI crawlers consume it yet — so treat it
as low-cost, optional, and complementary to the per-page discovery this plugin
already provides.

This plugin does not generate llms.txt. If you want one, publish a static file at your web root listing your
key pages with their ?output_format=md URLs, and keep it in sync with published
and retired content or it will point agents at missing pages.

What are taxonomy archive files?

For every public taxonomy term (categories, tags, custom taxonomies) the plugin
generates a Markdown file listing all published posts in that term with links and
excerpts. These are served automatically when an AI agent requests a taxonomy
archive URL. This lets agents navigate your site structure by exploring term listings,
not just individual posts.

What is the manifest.json file?

A manifest.json is created inside each post-type export folder (e.g.
wp-mfa-exports/post/manifest.json) whenever the downloadable bundle toggle is
on (it’s refreshed automatically before every bundle rebuild), or on demand via
–with-manifest or --incremental. It contains a registry of all exported
documents with content hashes and change tracking (new/modified/unchanged/deleted),
enabling RAG systems to identify what changed since the last export without
reprocessing all documents.

How does incremental export work?

Use wp markdown-agents generate --incremental to only re-export documents that
have changed since the last export. The plugin compares content hashes against the
previous manifest.json and skips unchanged posts. This also generates a
changes.json delta file listing new, modified, and deleted documents — your RAG
system can read this to know exactly what to re-embed.

How do I configure fields per post type?

In Settings → Markdown for Agents, each enabled post type has its own
“Field Configuration” section with two textareas:

• Frontmatter fields — meta or ACF fields added to the YAML frontmatter.
• Content fields — meta or ACF fields used as the body content. When set,
  post_content is automatically excluded.

Use dot notation for ACF group fields (e.g. clause_fields.clause_summary).
Plain meta keys work too (e.g. _yoast_wpseo_title). ACF relationship fields
are automatically converted to a list of post titles.

Can I customise the Markdown output?

Yes. Several filters are available:

• markdown_for_agents_pre_convert — filter HTML before conversion
• markdown_for_agents_post_convert — filter Markdown after conversion
• markdown_for_agents_frontmatter — modify frontmatter fields for a post
• markdown_for_agents_taxonomy_frontmatter — modify frontmatter fields for a taxonomy archive
• markdown_for_agents_serve_enabled — enable/disable serving for a specific post
• markdown_for_agents_serve_taxonomies — enable/disable serving for taxonomy archive pages
• markdown_for_agents_cache_headers — override the cache-related headers sent with the Markdown response (receives the access method since 1.6.1)
• markdown_for_agents_html_headers — modify or omit the Link and Vary: Accept headers added to HTML responses that have a Markdown alternate
• markdown_for_agents_file_generated — action fired after a file is written
• markdown_for_agents_file_deleted — action fired after a file is deleted

Can I let CDNs/full-page caches cache the Markdown responses?

By default the Markdown response is sent with Cache-Control: private, no-store, max-age=0 (plus X-LiteSpeed-Cache-Control, X-Accel-Expires and Vary: Accept, User-Agent). This is deliberate: the Markdown is negotiated on the same URL as the HTML page, so a shared cache that ignores or normalises Vary could otherwise store the Markdown variant and replay it to ordinary browsers expecting HTML.

The safe way to relax this is per access method, which the markdown_for_agents_cache_headers filter receives as its third argument (since 1.6.1). Requests via ?output_format=md can be cached publicly only when every cache preserves that query parameter in its key. Accept negotiation requires every cache to distinguish the negotiated variants. UA-only responses should remain private and bypass caches: keying on Accept alone does not separate them from browsers with the same Accept header. Map any header to an empty string to omit it entirely:

`

add_filter( ‘markdown_for_agents_cache_headers’, function ( array $headers, string $filepath = ”, string $access_method = ” ) {
// Only after verifying every cache key preserves output_format.
if ( ‘query-param’ === $access_method ) {
$headers[‘Cache-Control’] = ‘public, max-age=300’;
$headers[‘X-LiteSpeed-Cache-Control’] = ”;
$headers[‘X-Accel-Expires’] = ”;
}
// ‘accept-header’ and ‘ua’ responses share the HTML page’s URL — leave
// them private: a cache that ignores Vary (many do, including some
// managed WordPress hosts and CDNs) would replay Markdown to browsers.
return $headers;
}, 10, 3 );
`

The default values on $filepath and $access_method are deliberate: plugin versions before 1.6.1 pass fewer arguments to this filter, and required parameters would fatal every Markdown response with an ArgumentCountError if the snippet outlives a plugin downgrade (or is deployed ahead of the upgrade). With the defaults it is a safe no-op on older versions and activates automatically on 1.6.1+.

This filter governs only the cache-related headers listed above. The Content-Signal and X-Markdown-Source headers are sent separately and are unaffected (Content-Signal has its own markdown_for_agents_content_signal filter).

One caveat on caching the query-param URL: if a cache in front of the site is configured to ignore query strings when building cache keys, ?output_format=md collapses onto the page URL’s key and a public Markdown response could be served to browsers. The default therefore stays private; opt in only when you know your cache keys include the query string.

Why do HTML pages send `Vary: Accept` and a `Link` header?

Since 1.6.1, pages that have a Markdown alternate send two extra headers with the HTML response: Link: <…?output_format=md>; rel="alternate"; type="text/markdown" (protocol-level discovery, visible to HEAD requests and clients that do not parse HTML) and Vary: Accept (tells spec-correct shared caches not to replay a stored HTML variant to a client asking for text/markdown on the same URL — without it, a primed page cache answers agents with HTML before WordPress runs).

Both can be modified or removed with the markdown_for_agents_html_headers filter (map a header to an empty string to omit it). You may want to omit Vary: Accept if your cache layer refuses to store responses whose Vary lists anything beyond Accept-Encoding, as full-page caching then stops working entirely; in that case rely on the query-param URL for agent access instead.

How do I generate taxonomy archives via WP-CLI?

wp markdown-agents generate-taxonomies
wp markdown-agents generate-taxonomies --taxonomy=category
wp markdown-agents generate-taxonomies --dry-run