Apex US Regulatory Events Feed

openFDA Enforcement Report API Pagination: Getting Past the Skip Limit

A common first experience with openFDA: you write a loop with limit=1000&skip=N, it works for a while, then it errors out. You've hit the skip ceiling. Here's why, and three ways around it.

The limits

For openFDA endpoints, including drug/enforcement, device/enforcement and food/enforcement:

So skip alone can reach about 26,000 records. Drug enforcement alone is in that range and growing; device enforcement and recall data are larger. A naive loop silently stops short or fails.

Method 1: search_after (cursor paging)

openFDA supports cursor-style paging. When you sort results and pass search_after, the response includes a Link header pointing at the next page. Follow it until it disappears:

import requests, time

def fetch_all(url, params):
    rows = []
    while url:
        r = requests.get(url, params=params)
        r.raise_for_status()
        rows.extend(r.json()["results"])
        link = r.headers.get("Link")
        if link and 'rel="next"' in link:
            url = link[link.find("<")+1 : link.find(">")]
            params = None  # the next URL already carries the query
        else:
            url = None
        time.sleep(0.3)  # stay well inside rate limits
    return rows

rows = fetch_all("https://api.fda.gov/drug/enforcement.json",
                 {"limit": 1000, "sort": "report_date:asc",
                  "api_key": "YOUR_KEY"})

Check the openFDA docs for the exact parameter behaviour; implementations change. If Link doesn't appear for your query, fall back to Method 2.

Method 2: Date windows

Split the query by date so each slice stays below the skip ceiling:

import pandas as pd
def window_query(start, end):
    return f"report_date:[{start} TO {end}]"

all_rows = []
for year in range(2004, 2027):
    for half in [("0101", "0630"), ("0701", "1231")]:
        q = window_query(f"{year}{half[0]}", f"{year}{half[1]}")
        skip = 0
        while True:
            r = requests.get("https://api.fda.gov/drug/enforcement.json",
                             params={"search": q, "limit": 1000, "skip": skip})
            if r.status_code == 404:  # openFDA returns 404 for no matches
                break
            batch = r.json()["results"]
            all_rows.extend(batch)
            if len(batch) < 1000:
                break
            skip += 1000

Two notes:

Method 3: Bulk downloads (usually best)

If you want everything, skip the API. openFDA publishes the complete dataset as zipped JSON. The manifest at https://api.fda.gov/download.json lists every partition and its export date. For enforcement this is one or a few files, refreshed weekly. It's faster, uses no API quota, and can't silently miss pages.

Use the API for targeted queries (a firm, a classification, a date range) and bulk files for full loads.

De-duplicating after you page

Whichever method you use, check for duplicates on recall_number:

df = pd.DataFrame(all_rows).drop_duplicates("recall_number", keep="last")

Overlapping date windows, or records updated while you're paging, can produce duplicates.

The real problem: records change

Pagination gets you today's copy. But enforcement records get updated: status moves from Ongoing to Terminated, a product description is corrected, a classification is added. openFDA replaces the record, so yesterday's value is gone. If you need to know what changed, you have to:

  1. Save a full snapshot every week.
  2. Diff each snapshot against the last on recall_number, field by field.
  3. Store the diffs somewhere queryable.

That's the job the Apex US Regulatory Events Feed does. Each week it pulls the bulk files, cleans and de-duplicates them, and writes a weekly_changes table listing every record that was new, updated, terminated or removed that week, and which columns changed. It's delivered as CSV or via Delta Sharing for pandas, Spark and Power BI. It's a change history of FDA's published records, not a live recall-status service.

Free samples: https://apex-feed.pages.dev/#samples. Full feed: https://apex-feed.pages.dev.

Checklist

Source: U.S. FDA via openFDA. Apex is not affiliated with or endorsed by the U.S. Food and Drug Administration. API limits quoted may change; confirm on open.fda.gov.