Mandi price data looks deceptively simple: choose a commodity, fetch a number, and put it on a dashboard. The first time I worked through this dataset, the harder questions arrived immediately. Was the price per kilogram or per quintal? Was “modal” a typo? Did an empty result mean no trade, a spelling mismatch, or delayed reporting?

This article answers those questions while building a request against the Government of India’s current daily mandi-price resource. The data originates from AGMARKNET and is published by the Directorate of Marketing and Inspection through data.gov.in.

What the dataset actually represents

The catalog describes wholesale minimum, maximum, and modal prices reported daily by agricultural markets. “Modal price” means the price at which the largest volume of transactions is reported; it is not a predictive model and it is not necessarily the arithmetic average of the minimum and maximum.

  • state, district, and market locate the reporting mandi.

  • commodity, variety, and grade identify what was traded; comparing unlike varieties can produce misleading conclusions.

  • arrival_date is the reporting date carried by the record.

  • min_price, max_price, and modal_price are price observations. AGMARKNET documentation specifies DMI price units as rupees per quintal.

The request contract at a glance

  • Method: GET

  • Resource ID: 9ef84268-d588-465a-a308-a864a43d0070

  • Required: api-key and format

  • Formats: json, xml, or csv

  • Pagination: offset skips records and limit requests up to 1,000 records per call.

  • Filters: filters[state.keyword], filters[district], filters[market], filters[commodity], filters[variety], and filters[grade].

If you do not yet have a personal credential, create one using the data.gov.in API-key walkthrough. The portal’s shared demonstration key is intentionally limited and can be exhausted by other users.

Start with one small curl request

Enter the key without echoing it, request JSON, and begin with a single record. Keeping the first response small makes it easier to inspect the real schema before writing transformation code.

Terminalbash
read -rsp "data.gov.in API key: " DATA_GOV_API_KEY && printf '\n'

curl --fail-with-body --silent --show-error --get \
  'https://api.data.gov.in/resource/9ef84268-d588-465a-a308-a864a43d0070' \
  --data-urlencode "api-key=$DATA_GOV_API_KEY" \
  --data-urlencode 'format=json' \
  --data-urlencode 'offset=0' \
  --data-urlencode 'limit=1'
{
  "title": "Current Daily Price of Various Commodities from Various Markets (Mandi)",
  "total": 100000,
  "count": 1,
  "limit": "1",
  "offset": "0",
  "records": [
    {
      "state": "…",
      "district": "…",
      "market": "…",
      "commodity": "…",
      "variety": "…",
      "grade": "…",
      "arrival_date": "…",
      "min_price": "…",
      "max_price": "…",
      "modal_price": "…"
    }
  ]
}

What to inspect before going further

  • read -s prevents the key from being displayed as it is typed; the variable lasts only in the current shell unless exported.

  • --data-urlencode safely constructs the query string instead of relying on manual escaping.

  • --fail-with-body returns a failing exit status for HTTP errors but keeps the API’s error message visible.

  • The output above illustrates the documented envelope and common record fields; actual values, totals, and field availability change with the published data.

Filter by state and commodity

The filter names are not perfectly symmetrical: the current Swagger contract uses state.keyword for state, but plain field names for district, market, commodity, variety, and grade. Copy those names exactly.

Terminalbash
curl --fail-with-body --silent --show-error --get \
  'https://api.data.gov.in/resource/9ef84268-d588-465a-a308-a864a43d0070' \
  --data-urlencode "api-key=$DATA_GOV_API_KEY" \
  --data-urlencode 'format=json' \
  --data-urlencode 'offset=0' \
  --data-urlencode 'limit=10' \
  --data-urlencode 'filters[state.keyword]=Maharashtra' \
  --data-urlencode 'filters[commodity]=Onion'
{
  "count": 10,
  "records": [
    { "state": "Maharashtra", "commodity": "Onion", "market": "…", "modal_price": "…" }
  ]
}

Why exact filter values matter

  • The bracketed names are literal query parameters; curl encodes the brackets and spaces correctly.

  • Filter values follow the publisher’s spelling and capitalization. An apparently reasonable alias may return zero rows.

  • A count smaller than limit is normal. It reports records in this page, not necessarily every matching record.

  • Do not assume the first page contains the newest observations unless the resource contract explicitly documents a sort order.

Build a reusable Python fetcher

For scheduled ingestion, preserve the raw strings first and parse them deliberately. Price fields often arrive as strings, and converting them straight to binary floating point can quietly complicate money calculations.

fetch_mandi_prices.pypython
import os
from datetime import datetime
from decimal import Decimal, InvalidOperation
 
import requests
 
RESOURCE_ID = "9ef84268-d588-465a-a308-a864a43d0070"
API_URL = f"https://api.data.gov.in/resource/{RESOURCE_ID}"
 
 
def as_decimal(value: object) -> Decimal | None:
    try:
        return Decimal(str(value))
    except (InvalidOperation, TypeError, ValueError):
        return None
 
 
def fetch_prices(state: str, commodity: str, limit: int = 100) -> list[dict]:
    api_key = os.environ.get("DATA_GOV_API_KEY")
    if not api_key:
        raise RuntimeError("DATA_GOV_API_KEY is not set")
 
    response = requests.get(
        API_URL,
        params={
            "api-key": api_key,
            "format": "json",
            "offset": 0,
            "limit": min(max(limit, 1), 1000),
            "filters[state.keyword]": state,
            "filters[commodity]": commodity,
        },
        timeout=30,
    )
    response.raise_for_status()
    payload = response.json()
 
    rows = []
    for record in payload.get("records", []):
        row = dict(record)
        row.update(
            {
                "min_price_decimal": as_decimal(record.get("min_price")),
                "max_price_decimal": as_decimal(record.get("max_price")),
                "modal_price_decimal": as_decimal(record.get("modal_price")),
                "retrieved_at_utc": datetime.now().astimezone().isoformat(),
            }
        )
        rows.append(row)
    return rows
 
 
if __name__ == "__main__":
    for row in fetch_prices("Maharashtra", "Onion", limit=10):
        print(row["arrival_date"], row["market"], row["modal_price_decimal"])

Fetch one filtered page and parse price values as exact decimals.

Design choices worth keeping

  • requests.get(..., params=...) handles query encoding, while timeout=30 prevents an unlimited network wait.

  • min(max(limit, 1), 1000) enforces the current API contract’s documented range at the client boundary.

  • Decimal(str(value)) avoids introducing an extra binary-float approximation when parsing reported prices.

  • Malformed or missing price values become None; that preserves the row and lets downstream validation decide how to handle it.

  • retrieved_at_utc records collection time separately from arrival_date, which belongs to the source record.

Paginate without losing your place

For a larger extract, request pages sequentially and advance offset by the number of records actually returned. Stop when the API returns no records or when the collected count reaches a trustworthy total value.

pagination_loop.pypython
offset = 0
page_size = 1000
 
while True:
    response = requests.get(
        API_URL,
        params={
            "api-key": api_key,
            "format": "json",
            "offset": offset,
            "limit": page_size,
        },
        timeout=30,
    )
    response.raise_for_status()
    payload = response.json()
    records = payload.get("records", [])
 
    if not records:
        break
 
    process(records)
    offset += len(records)

A defensive pagination pattern for the resource endpoint.

Operational lessons in this loop

  • The loop advances by len(records), not blindly by 1,000, so a short page does not create a gap.

  • process(records) is intentionally a placeholder for your validated storage or transformation function.

  • A production job should also implement bounded retries with backoff for transient failures and checkpoint its last completed offset.

  • If the dataset changes during a long extraction and the API offers no stable ordering, offsets alone cannot guarantee a snapshot-consistent result.

Freshness and data-quality checks

  • Stale `arrival_date`: report it visibly and compare it with the expected market schedule; do not replace it with today’s date.

  • Minimum above maximum: quarantine the row for review rather than silently swapping values.

  • Modal price outside the range: flag it as suspicious because the modal observation should normally sit between reported minimum and maximum.

  • Missing market or variety: retain the raw row and mark the dimension unknown; dropping it can distort aggregates.

  • Duplicate-looking rows: define a business key from date, location, commodity, variety, and grade before deduplicating.

  • Unit conversion: divide rupees per quintal by 100 only when you intentionally need rupees per kilogram, and label the converted value.

Troubleshooting the endpoint

  • `Authorization field missing`: include the api-key query parameter; a resource UUID alone is not sufficient.

  • `Rate limit exceeded`: stop retrying aggressively. Use your personal key, respect backoff, and reduce unnecessary duplicate calls.

  • Zero filtered records: remove filters one at a time and copy valid spellings from an unfiltered response.

  • HTTP 400: check format, numeric pagination values, maximum limit, and exact parameter names against the current Swagger page.

  • Records are older than expected: distinguish a working API from fresh source reporting; an HTTP 200 does not guarantee today’s data.

  • Key appears in logs: rotate it and redact query strings at the HTTP client, proxy, monitoring, and exception-reporting layers.

Official references