How to Send Email From Splunk With Volanea

Send email from Splunk: the practical architecture

To send email from Splunk through Volanea, start with a Splunk alert generated by a saved search, send its webhook payload to a relay you control, and have that relay call Volanea’s email API. This pattern is deliberately small: Splunk detects the condition, the relay translates and secures it, and Volanea delivers the message.

That middle step matters. Splunk’s native webhook alert action posts Splunk’s alert-event envelope, while an email API expects a message payload with recipients, a sender, a subject, and content. A relay is therefore not needless infrastructure; it is the boundary that converts monitoring data into an intentional, safe customer or operator notification.

Splunk does provide an outbound webhook alert action. It does not require a native Volanea app, and Volanea does not provide a Splunk marketplace plug-in. The integration described here uses the capability Splunk actually has: an alert action attached to a saved search. It is suitable for incident notifications, failed-job summaries, security escalation notices, SLA warnings, and internal workflow emails.

The high-level request path is:

1. A scheduled saved search in Splunk finds matching events.
2. Its alert condition is met, such as “number of results is greater than 0.”
3. Splunk’s webhook alert action POSTs a JSON alert payload to a relay endpoint.
4. The relay validates the request, maps result fields into an email, and calls Volanea.
5. The relay records an idempotency key and returns a fast successful HTTP response to Splunk.

This approach keeps a mail-provider credential out of Splunk’s alert payload and gives you a controlled place to format, suppress, deduplicate, route, and audit alerts.

The Splunk trigger that starts the email

The trigger is a saved search alert. A saved search can run on a schedule or in real time; when its alert condition evaluates to true, Splunk executes configured alert actions. One of those actions can be a webhook.

For a concrete example, imagine a scheduled saved search that checks an application index every five minutes:

index=payments sourcetype=api-gateway status>=500
| stats count as error_count values(host) as hosts values(request_id) as request_ids by service
| where error_count >= 10

Save the search as an alert. Set it to run every five minutes, and use an alert condition based on the number of results being greater than zero. In this example, each result row represents one affected service; you can configure the alert’s triggering behavior according to whether you want a single summary notification or one notification per matching result.

That distinction changes the email experience:

• One alert for the search run is usually best for an on-call or incident mailbox. It avoids a separate email for every host or event.
• One alert per result can be useful when each result belongs to a different service owner and the relay routes messages by a field such as service or team_email.
• A throttled alert is essential for noisy conditions. It prevents an unchanged failure from creating an email every time the saved search runs.

Before connecting an email provider, make the search output email-ready. Prefer explicit, stable fields such as service, error_count, hosts, environment, runbook_url, and severity. Do not make the relay parse _raw unless there is no alternative. Parsing raw log text after an incident has started is fragile, difficult to test, and likely to expose more data than the email needs.

Design the search result as a notification contract

Treat the final row produced by the search as a contract between Splunk and your relay. For example:

index=payments sourcetype=api-gateway status>=500
| stats count as error_count values(host) as hosts by service environment
| where error_count >= 10
| eval severity="critical"
| eval runbook_url="https://runbooks.example.com/payments-5xx"
| table service environment severity error_count hosts runbook_url

The table command is useful because it makes the intended output obvious. If a later editor changes the search, a test alert can quickly reveal whether the fields expected by the relay still exist.

Avoid putting recipient addresses directly in ordinary event data. If routing is dynamic, map a controlled value such as service to an allowlisted recipient group in the relay. A log field can be malformed or attacker-controlled; it should not be able to make your alerting system email arbitrary external addresses.

What Splunk sends to the relay

Splunk’s webhook alert action makes an HTTP POST and sends a JSON alert payload. The payload is an alert envelope: it includes alert metadata and, when the alert is configured around result data, a result object containing fields from the search result.

A representative payload from the saved-search example looks like this:

{
  "result": {
    "service": "payments-api",
    "environment": "production",
    "severity": "critical",
    "error_count": "27",
    "hosts": "[\"pay-01\", \"pay-02\"]",
    "runbook_url": "https://runbooks.example.com/payments-5xx"
  },
  "sid": "scheduler_admin_search_RMD5f2b6f8e31f9b0f7_at_1720000000_12345",
  "search_name": "Payments API elevated 5xx rate",
  "owner": "admin",
  "app": "search",
  "results_link": "https://splunk.example.com/app/search/search?sid=scheduler_admin_search_RMD5f2b6f8e31f9b0f7_at_1720000000_12345",
  "search": "index=payments sourcetype=api-gateway status>=500 ...",
  "alert_type": "number of events"
}

The exact values, optional properties, and shape of result fields depend on the saved search, the alert’s trigger configuration, and your Splunk deployment. In particular, multivalue search fields may arrive as strings rather than as JSON arrays. Your relay should normalize them rather than assuming every field has one type.

The fields that are most useful for a relay are usually:

Configure the webhook URL to point to your relay, for example https://alerts.example.com/splunk/volanea. Use HTTPS. The relay should be reachable from Splunk Cloud or from the network where your Splunk Enterprise instance runs, but it should not expose a general-purpose unauthenticated email-sending endpoint.

Why a relay is required instead of a direct API call

A direct webhook from Splunk to an email API is tempting because it appears to remove a component. In practice, the payload contract is wrong for direct delivery. Splunk posts its own alert envelope; Volanea needs an email-send request. The webhook action is for delivering an alert payload, not for performing arbitrary per-field JSON transformation into every third-party API schema.

A relay solves four separate problems:

1. Payload transformation. It turns result.service, result.error_count, and results_link into the recipient, subject, text, and HTML fields of an email.
2. Credential isolation. It keeps the Volanea API key in server-side secret storage rather than in a Splunk configuration field or webhook URL.
3. Delivery control. It can suppress repeats, use an idempotency store, route by service, and distinguish test alerts from production incidents.
4. Operational observability. It can log Volanea response identifiers without logging message bodies, API keys, or sensitive raw events.

The relay can be a small Node.js service, an existing internal integration service, a serverless function behind an API gateway, or an automation workflow with secure secret handling. For production incident mail, code you own is often easier to test and reason about than a chain of opaque transformations.

If you choose Zapier or Make as middleware, use Splunk’s webhook to invoke a Catch Hook/custom webhook endpoint, then map the incoming result fields into an authenticated HTTP request to Volanea. Keep the Volanea credential in that platform’s connection or secret facility, not in a mapped field. The same deduplication concern still applies: automation workflows may receive retries or replayed webhook deliveries.

Working relay: map a Splunk alert into a Volanea email

The following Node.js example accepts the Splunk POST, verifies a shared webhook secret at the edge, maps the alert payload, and sends an email through Volanea. It uses environment variables for configuration and sends a deterministic Idempotency-Key based on the Splunk search ID.

Review the current request fields and authentication details in the email API reference and setup guides before deploying, especially if your Volanea account uses a different API version or you want to use templates instead of inline HTML.

// server.mjs
import crypto from "node:crypto";
import express from "express";

const app = express();
app.use(express.json({ limit: "256kb" }));

const {
  // Protect this route at your API gateway or reverse proxy.
  // The relay compares this value with the incoming shared secret.
  SPLUNK_WEBHOOK_SECRET,

  // Keep this only in server-side secret storage.
  VOLANEA_API_KEY,
  VOLANEA_FROM = "Splunk Alerts <alerts@example.com>",
  ALERT_TO = "oncall@example.com"
} = process.env;

function sameSecret(received, expected) {
  if (!received || !expected) return false;
  const a = Buffer.from(received);
  const b = Buffer.from(expected);
  return a.length === b.length && crypto.timingSafeEqual(a, b);
}

function escapeHtml(value) {
  return String(value ?? "")
    .replaceAll("&", "&amp;")
    .replaceAll("<", "&lt;")
    .replaceAll(">", "&gt;")
    .replaceAll('"', "&quot;")
    .replaceAll("'", "&#039;");
}

function text(value, fallback = "Not available") {
  return value === undefined || value === null || value === ""
    ? fallback
    : String(value);
}

app.post("/splunk/volanea", async (req, res) => {
  // Have your gateway/proxy place the validated secret in this header.
  // Do not put VOLANEA_API_KEY in Splunk or in this request.
  const suppliedSecret = req.get("x-splunk-webhook-secret");
  if (!sameSecret(suppliedSecret, SPLUNK_WEBHOOK_SECRET)) {
    return res.status(401).json({ error: "unauthorized" });
  }

  const payload = req.body ?? {};
  const result = payload.result ?? {};

  // Field mapping from Splunk webhook JSON to an email message.
  const service = text(result.service, "unknown service");
  const environment = text(result.environment, "unknown environment");
  const severity = text(result.severity, "warning").toUpperCase();
  const errorCount = text(result.error_count, "0");
  const hosts = text(result.hosts, "No host list provided");
  const runbookUrl = text(result.runbook_url, "");
  const searchName = text(payload.search_name, "Splunk alert");
  const resultsLink = text(payload.results_link, "");

  if (!payload.sid) {
    return res.status(400).json({ error: "missing Splunk sid" });
  }

  const subject = `[${severity}] ${service} in ${environment}: ${errorCount} errors`;
  const html = `
    <h2>${escapeHtml(searchName)}</h2>
    <p><strong>Service:</strong> ${escapeHtml(service)}<br>
       <strong>Environment:</strong> ${escapeHtml(environment)}<br>
       <strong>Severity:</strong> ${escapeHtml(severity)}<br>
       <strong>Error count:</strong> ${escapeHtml(errorCount)}<br>
       <strong>Hosts:</strong> ${escapeHtml(hosts)}</p>
    ${runbookUrl ? `<p><a href="${escapeHtml(runbookUrl)}">Open runbook</a></p>` : ""}
    ${resultsLink ? `<p><a href="${escapeHtml(resultsLink)}">Open Splunk results</a></p>` : ""}
  `;
  const plainText = `${searchName}\n\nService: ${service}\nEnvironment: ${environment}\nSeverity: ${severity}\nError count: ${errorCount}\nHosts: ${hosts}\n${runbookUrl ? `Runbook: ${runbookUrl}\n` : ""}${resultsLink ? `Splunk results: ${resultsLink}` : ""}`;

  // Use a durable database/Redis uniqueness record here in production.
  // Do not rely only on an in-memory Set after a process restart.
  const volaneaResponse = await fetch("https://api.volanea.com/v1/emails", {
    method: "POST",
    headers: {
      "Authorization": `Bearer ${VOLANEA_API_KEY}`,
      "Content-Type": "application/json",
      "Idempotency-Key": `splunk:${payload.sid}`
    },
    body: JSON.stringify({
      from: VOLANEA_FROM,
      to: [ALERT_TO],
      subject,
      html,
      text: plainText,
      tags: ["splunk", environment, severity.toLowerCase()]
    })
  });

  const responseBody = await volaneaResponse.text();
  if (!volaneaResponse.ok) {
    console.error("Volanea send failed", {
      status: volaneaResponse.status,
      sid: payload.sid,
      responseBody
    });
    return res.status(502).json({ error: "email provider rejected request" });
  }

  console.info("Splunk email accepted", {
    sid: payload.sid,
    searchName,
    providerResponse: responseBody
  });
  return res.status(202).json({ accepted: true });
});

app.listen(3000, () => console.log("relay listening on :3000"));

The important mapping is not hidden in a template: payload.result.service becomes the service name in the subject and body, payload.result.error_count becomes the count, and payload.results_link becomes the responder’s link back to Splunk. The sid becomes the idempotency identity.

Do not use the search property in the outgoing email by default. Saved searches can reveal index names, query details, account identifiers, or other internal context that recipients do not need. Send a results link to authorized responders instead.

Authentication and secret handling

The Volanea API key should not live on the Splunk side in this architecture. It belongs in the relay’s server-side secret manager: a cloud secret manager, encrypted deployment secret, or platform environment secret with access limited to the relay’s runtime identity.

This is more than a tidy separation of duties. A Volanea API key authorizes email sending. Putting it in a client-visible browser configuration, a query parameter, a saved-search string, a webhook body, a source repository, or a broadly readable Splunk configuration turns a monitoring integration into an account-takeover risk.

Use two different credentials:

• Splunk-to-relay authentication: a separately generated shared secret, gateway token, mTLS identity, signed request scheme, or private-network control. This only grants access to the one relay route.
• Relay-to-Volanea authentication: the Volanea API key, stored only in server-side secret storage and sent by the relay over HTTPS.

The code above models a shared secret as x-splunk-webhook-secret, but your ingress layer must implement the exact header or authentication mechanism supported by your Splunk webhook configuration. Do not assume that a URL path containing a random token is sufficient authentication. A reverse proxy or API gateway can validate the Splunk-facing credential, then forward the request to the application with a trusted internal header.

Keep sender and recipient policy in configuration

Keep VOLANEA_FROM and the default ALERT_TO as server-side configuration too. The from address must be a sender/domain your Volanea account is authorized to use. The recipient can be an on-call distribution list, a ticket intake mailbox, or an incident-management address.

For routing by service, create an allowlist in code or a database:

const recipientsByService = {
  "payments-api": "payments-oncall@example.com",
  "identity-api": "identity-oncall@example.com"
};
const to = recipientsByService[result.service] ?? "platform-oncall@example.com";

This is safer than accepting result.recipient_email from log data. It also makes ownership changes reviewable without editing dozens of searches.

Make alert emails useful rather than noisy

Email is a durable notification channel, not a substitute for a paging system. Use it for alerts where an inbox is the right place to coordinate, document, or escalate. For a time-critical service outage, pair it with the organization’s on-call paging path rather than hoping an email is seen immediately.

Good Splunk alert emails answer five questions in the first screenful:

• What happened?
• Which service and environment are affected?
• How severe is it, and what measured threshold was crossed?
• Where can the responder investigate?
• What is the first remediation or runbook step?

A subject such as [CRITICAL] payments-api in production: 27 errors is actionable. A subject such as Splunk Alert: number of events requires opening the email and interpreting monitoring internals before anyone knows whether action is needed.

Prefer summaries for bursts

A five-minute search can repeatedly detect the same incident. Throttling on the Splunk alert reduces the number of action invocations, but the relay should also protect the mailbox. A useful policy is to create one email when an incident starts, suppress identical alerts for a defined window, and send a recovery or periodic summary only when there is meaningful new information.

The deduplication identity should include enough context to distinguish real incidents. A raw sid is useful for one search execution, while a durable incident key might combine search_name, service, environment, and a normalized severity. Store a hash of that key with an expiry in Redis or a database. Do not use the entire raw event as a key; variable timestamps and IDs will defeat suppression.

Keep sensitive data out of the template

Logs often contain user IDs, IP addresses, session tokens, request bodies, or error stack traces. An email can be forwarded outside the original operational context and may remain in mailboxes longer than the source data remains in Splunk. Build a deliberately small notification schema and link authorized responders to Splunk for deeper investigation.

Use text and HTML versions of the email. The plain-text content improves accessibility and provides a readable fallback when a mail client blocks HTML. Escape every field injected into HTML; a service name or raw field should never be trusted as markup.

When this breaks

Every integration has two delivery hops: Splunk to the relay, then the relay to Volanea. Diagnose them separately. A saved search can be healthy while its webhook cannot reach the relay; the relay can accept the webhook while the email send fails because of sender authorization, recipient policy, or an API error.

Retries and duplicate emails

Do not assume a webhook action is delivered exactly once. A timeout, connection reset, process crash after sending, alert rerun, or retry behavior in the Splunk-to-relay path can result in the same logical alert reaching the relay more than once. If the relay sends the email before persisting a deduplication record, it can create duplicate notices.

Use a durable idempotency record before or alongside the provider call. The key can start with sid, but test whether a repeated Splunk delivery retains the same search ID in your environment. For broader incident suppression, use an alert fingerprint as described above. Return a successful response only after the relay has safely accepted responsibility for the alert.

Webhook timeouts

The relay must respond quickly. Do not wait for slow enrichment calls, an external CMDB lookup, or a long retry loop before acknowledging the webhook. A slow response can cause Splunk to regard the action as failed even if the relay eventually sends an email, which is a classic cause of duplicate sends.

A robust design writes the validated alert to a queue or database, returns a 2xx response, and lets a worker call Volanea. If you keep the synchronous version shown above, set conservative outbound timeouts, log the provider response, and measure end-to-end latency. Make sure your gateway timeout is aligned with your application timeout rather than terminating the request first.

Missing or differently typed payload fields

The result object is not guaranteed to contain every field for every alert design. A search that emits no rows, a change in the final table, a different trigger mode, or differences in available alert-result context can leave result.service, error_count, or results_link absent. Multivalue fields can also arrive serialized as text.

Validate required fields at the relay boundary. For a critical routing field such as service, either reject the request into a dead-letter path or route it to a platform fallback recipient; do not silently send a vaguely titled email to an arbitrary address. For display-only fields, use a visible fallback such as “Not available” so an alert template does not conceal the data-quality problem.

Volanea rejection and delivery failures

A successful response from your relay does not mean a recipient has read the message. It means the relay accepted the Splunk alert, and a successful Volanea API response means the provider accepted the send request. Monitor API response codes, provider message identifiers, bounce events where appropriate, and sender-domain authentication.

Common provider-side causes include an unauthorized sender address, malformed recipient data, invalid JSON, rate limits, or a temporarily unavailable API. Log the HTTP status and provider request identifier, but redact authorization headers and avoid storing full message content in general application logs.

Test the integration before relying on it

Start in a non-production environment with a test sender domain and a controlled recipient mailbox. Create a saved search that always produces a known test result, invoke the webhook, and inspect the relay’s structured logs. Then test the negative cases deliberately.

A practical test checklist is:

1. Confirm the saved search produces exactly the fields the relay expects.
2. Confirm the webhook reaches the relay over HTTPS and fails closed without valid relay authentication.
3. Confirm the relay sends a readable plain-text and HTML email through Volanea.
4. Replay the same payload twice and verify the recipient receives only one email.
5. Remove service or error_count from a test payload and verify your fallback or dead-letter behavior.
6. Force a provider failure and verify that the relay logs an actionable error without exposing credentials.
7. Trigger a burst of matching results and verify Splunk throttling plus relay suppression meet the intended notification policy.

Keep a sanitized sample webhook payload in your test suite. It lets you regression-test field mapping whenever the saved search changes, without needing to wait for a real production incident. Include a test for unusual characters in service and hosts to confirm HTML escaping works.

Operating the integration over time

The first deployment is only the beginning. Saved searches evolve, services are renamed, teams change ownership, and alert volumes change as systems grow. Make the relay configuration versioned and observable so those operational changes do not become hidden email-routing failures.

Track at least these metrics: received Splunk webhooks, authentication failures, schema-validation failures, suppressed duplicates, Volanea API successes, Volanea API failures, and relay processing latency. Alert on a sustained discrepancy between received webhooks and accepted email requests. That discrepancy is often the first signal that a sender credential expired, a gateway policy changed, or a search output was edited.

Review sender identity and DNS authentication before moving incident mail to production. A properly authenticated sending domain gives operational emails a better chance of landing where responders expect them, while a vague sender or an unverified domain can turn an otherwise correct monitoring workflow into a deliverability problem.

For cost planning, consider both normal alert volume and failure bursts. Email volume is often low until an incident makes it high, so include throttling and suppression in your estimates of transactional email sending costs. The cheaper system is not the one that emits the most notifications; it is the one that reliably sends the few messages that lead to action.

Conclusion

The reliable way to send email from Splunk with Volanea is to use Splunk’s saved-search alert and webhook action as the trigger, then translate that webhook through a secure relay. The relay gives you explicit field mapping, server-side API-key handling, fast acknowledgements, deduplication, and a clear audit boundary.

Build the saved search output as a stable notification contract, keep recipient routing allowlisted, and test duplicate and missing-field cases before an incident tests them for you. With those controls in place, Splunk alert data can become concise, deliverable, and actionable email instead of an unstructured stream of monitoring noise.

FAQ

Can Splunk send directly to Volanea without a relay?

Not reliably for a mapped transactional email workflow. Splunk’s native webhook sends a Splunk alert payload, while Volanea needs an email-send request. A relay or automation middleware performs the necessary transformation and protects the Volanea credential.

What exactly triggers the email in Splunk?

A saved search alert triggers it. For example, a scheduled search can run every five minutes and execute its webhook action when the number of search results is greater than zero.

Where should the Volanea API key be stored?

Store it in the relay’s server-side secret manager or protected runtime environment. Do not place it in Splunk search text, the webhook payload, a URL, browser-visible configuration, or a source repository.

How do I prevent duplicate Splunk alert emails?

Use Splunk alert throttling and relay-side durable idempotency. Store a key derived from the Splunk sid or a stable incident fingerprint before allowing a second send for the same logical alert.

What should the relay do when a result field is missing?

Validate fields required for routing, such as service, and send failures to a controlled fallback or dead-letter path. For display-only fields, show an explicit fallback rather than silently producing a misleading email.