Splunk API: Creating Search Jobs, Polling Them, and Not Melting the Cluster

The Splunk REST API models a search as a job: you create it with a POST, poll until it reports done, then fetch results from a separate endpoint. Understanding that three-step shape explains almost every question people have about it.

The API is comprehensive and the documentation is large, which makes it hard to find the thirty lines you actually need. The core of it is the search job lifecycle, and once that is clear the rest is looking up endpoint names.

Authentication
The search job lifecycle
Export mode: skipping the dance
Not melting the search head
Using the SDK instead
How this fits the rest of the stack
FAQ

Authentication

Two options. A session token obtained by posting credentials, or a long-lived authentication token created in the interface. Prefer the token for anything automated.

export SPLUNK_HOST=https://splunk.example.com:8089
export SPLUNK_TOKEN=eyJraWQiOiJzcGx1bmsu...

# Verify it works
curl -k -H "Authorization: Bearer $SPLUNK_TOKEN" \
  "$SPLUNK_HOST/services/authentication/current-context?output_mode=json"

Note the port. The management API is on 8089, not the 8000 you use for the web interface. Pointing a script at 8000 produces confusing HTML responses instead of the API errors you expected, and it is the most common first mistake.

The output mode parameter is worth setting on every request. Splunk defaults to XML, and adding it to each call is far less irritating than parsing XML because you forgot.

On the insecure flag: many Splunk deployments use self-signed certificates, which is why it appears in every example. Skipping verification is fine against an internal host you control and is not fine as a permanent habit - point at the CA bundle instead once you have one.

The search job lifecycle

Three steps: create, poll, fetch.

# 1. Create the job - note the required search prefix
SID=$(curl -k -s -H "Authorization: Bearer $SPLUNK_TOKEN" \
  -d "search=search index=web status=500 earliest=-1h" \
  -d "output_mode=json" \
  "$SPLUNK_HOST/services/search/jobs" | jq -r '.sid')

echo "job: $SID"

That literal search keyword at the start of the query string is required unless the search begins with a generating command such as a metadata or rest lookup. Omitting it is the second most common mistake, and the error is not obvious.

# 2. Poll until done
while true; do
  DONE=$(curl -k -s -H "Authorization: Bearer $SPLUNK_TOKEN" \
    "$SPLUNK_HOST/services/search/jobs/$SID?output_mode=json" \
    | jq -r '.entry[0].content.isDone')
  [ "$DONE" = "true" ] && break
  sleep 2
done

# 3. Fetch results
curl -k -s -H "Authorization: Bearer $SPLUNK_TOKEN" \
  "$SPLUNK_HOST/services/search/jobs/$SID/results?output_mode=json&count=0" \
  | jq '.results'

Setting count to zero means unlimited. The default is a small number, and quietly getting the first hundred rows of a search you believed returned thousands is a bug that survives testing.

The job status response carries more than the done flag - a progress fraction, the event and result counts, the scan count, and the run duration. If you are polling anyway, surfacing the progress makes a long-running job far less opaque.

Export mode: skipping the dance

For large result sets, the export endpoint streams results as they are produced rather than making you poll and then fetch.

curl -k -s -H "Authorization: Bearer $SPLUNK_TOKEN" \
  -d "search=search index=web earliest=-24h" \
  -d "output_mode=csv" \
  "$SPLUNK_HOST/services/search/jobs/export" \
  > results.csv

This is the right tool when you want everything and intend to process it as a stream. No job to poll, no artifact left on the search head, and memory usage stays bounded because you are not materialising the whole result set.

The trade-off is that you cannot check status, cancel cleanly, or re-fetch the results later - if the connection drops, you start again. For interactive or resumable work, the job lifecycle is the better fit. For a scheduled extract, export is simpler and lighter.

There is also a blocking mode on job creation, where the request does not return until the search is complete. It is convenient for short searches and a bad idea for anything that might run for minutes, since you are holding a connection open the whole time.

Not melting the search head

This is the part that gets automation authors in trouble with their Splunk administrator. Every search consumes resources on the search head, and an API client can create searches far faster than a human can.

Always bound the time range. Put earliest and latest in the search itself, or pass them as parameters. An unbounded search over a large index is the single most expensive mistake available.
Filter at the index level. Specify the index and sourcetype. Searching across everything to find one sourcetype scans enormously more data.
Prefer summarising commands. Return aggregates using stats or timechart rather than pulling raw events and aggregating client-side. Less data crosses the wire and less work happens on the head.
Reuse job results. A finished job’s results remain fetchable until its time-to-live expires. Do not re-run an identical search because you needed the results twice.
Clean up. Delete jobs you no longer need rather than leaving them to expire.
Limit concurrency. There is a cap on concurrent searches per user and per instance. Exceeding it queues your searches and degrades everyone else’s.

# Cancel or delete a job
curl -k -X DELETE -H "Authorization: Bearer $SPLUNK_TOKEN" \
  "$SPLUNK_HOST/services/search/jobs/$SID"

# What is currently running under this account
curl -k -s -H "Authorization: Bearer $SPLUNK_TOKEN" \
  "$SPLUNK_HOST/services/search/jobs?output_mode=json" \
  | jq -r '.entry[] | "\(.name) \(.content.dispatchState)"'

One thing that catches people: listing jobs returns everything visible to the account, which on a shared search head can be hundreds of entries including other people’s. If your script expects only its own jobs, filter by the SIDs you created rather than assuming.

Using the SDK instead

Official SDKs exist for several languages and wrap the polling loop, authentication, and result parsing. For anything more than a couple of scripts, they are worth the dependency.

import splunklib.client as client
import splunklib.results as results

service = client.connect(
    host="splunk.example.com",
    port=8089,
    splunkToken=TOKEN,
)

job = service.jobs.create(
    "search index=web status=500 | stats count by host",
    earliest_time="-1h",
    latest_time="now",
)

while not job.is_done():
    time.sleep(1)

for row in results.JSONResultsReader(job.results(output_mode="json")):
    if isinstance(row, dict):
        print(row)

job.cancel()

The SDK still creates a job and still polls - it has not changed the model, only hidden the boilerplate. Understanding the underlying lifecycle remains worthwhile, because when something goes wrong the error you get back is about the job, and the fix is usually in the search rather than the client.

Note the reader yielding both dictionaries and message objects. Filtering for dictionaries is not defensive programming; Splunk genuinely interleaves diagnostic messages with results, and skipping that check produces a confusing crash on the first search that emits a warning.

How this fits the rest of the stack

Log-search automation is downstream of getting logs in the first place, and for a deployed application that means the build log and the runtime log being somewhere you can reach. On RunxBuild both sit with the deploy that produced them, alongside metrics and rollback to the previous version. The RunxBuild hosting calculator shows the service and its managed database as separate line items, so the system generating those logs has a known cost.

Useful related references:

FAQ

How do I run a Splunk search via the API?

Post to the search jobs endpoint to create a job, poll the job endpoint until it reports done, then fetch from the results endpoint. Remember the literal search keyword at the start of the query string unless your search begins with a generating command.

What port does the Splunk REST API use?

Port 8089, the management port - not 8000, which serves the web interface. Pointing a script at 8000 returns HTML instead of API responses, which is the most common first mistake people make.

Why does my Splunk API search return only some results?

The results endpoint has a default row limit. Pass a count of zero to request all rows. Silently receiving the first page of a much larger result set is a bug that survives testing, because the response looks perfectly valid.

What is the difference between search jobs and export?

A job is created, polled, and then fetched, and its results stay retrievable until the job expires. Export streams results as they are produced with no job to manage, which suits large one-off extracts but cannot be checked, cancelled cleanly, or re-fetched.

How do I avoid overloading the Splunk search head?

Bound every search with an explicit time range, filter by index and sourcetype, return aggregates using stats or timechart rather than raw events, reuse finished job results instead of re-running searches, delete jobs when done, and limit how many searches you dispatch concurrently.