Keeping AI Requests From Disappearing

A request that never returns

Reports of this problem are easy to find. A call to a language model runs for a minute or two, sometimes longer, and returns nothing. There is no response, and no error that explains the delay. The client gives up or the connection closes, and the outcome cannot be determined.

Latency is not the core issue. A slow request is manageable as long as you know it is still running. The difficulty is that the request becomes invisible: it may have completed, it may have failed, or it may still be running on a server you can no longer reach. This condition is unknown request state, and it is the problem this guide addresses.

Hanging, failure, and unknown state

These three outcomes are often treated as one, but they behave differently. A failure returns a status code your code can act on. A request that hangs is still open, so you can keep waiting or set a deadline. Unknown state is the hard case: the connection has ended, and you cannot tell whether the work completed.

A single try/catch around the call hides the distinction. When the call throws after a long wait, the handler cannot separate a provider error from a dropped connection from a request that already succeeded.

• Failure — the request returned an error you can handle.
• Hanging — the request is still open and the result is pending.
• Unknown — the connection ended and the outcome is undetermined.

Why an unknown outcome is the real cost

An error is a decision the system makes for you. Unknown state moves that decision to a person, usually the user, at the least convenient moment.

Retrying becomes a guess. If the first request finished, a retry repeats the work and the cost. If it did not, skipping the retry discards the result. The application has no basis to choose, so it shows a spinner or asks the user to try again.

This matters more for AI features, where the generated output is often the product. A lost or duplicated result is not a minor backend issue. It is the user's work, with no reliable record of what happened to it.

Common symptoms

The reports describe the same situation in different forms. Each points to a missing piece: there is no durable record of the request, so there is nothing to inspect after the fact.

• A call that runs for minutes and then throws a generic timeout.
• A stream that opens, falls silent, and never closes.
• A 504 or a closed socket with no response body.
• A retry that may have started a second, equally expensive call.
• Logs that record the start of a request but not its outcome.
• A support request that cannot be traced to a single call.

Where ReqRun fits

Keeping a request from disappearing should not require building durable request tracking by hand. ReqRun provides it as a hosted service.

ReqRun sits in front of your OpenAI-compatible call. You send the request to ReqRun rather than to the provider directly. It assigns a durable rr_ identifier as soon as the request is accepted, runs the call on its own infrastructure, records each attempt, and reports the outcome through a status endpoint.

Because the request lives in the service rather than in a single connection, a hang is no longer fatal. The connection can drop, the function can recycle, or the user can refresh, and the request remains available to look up.

Request states

Status is available from the moment a request is accepted, not only when it completes. As soon as ReqRun receives the request, it has an identifier and a state, before the model returns any output.

From there the request moves through a small, fixed set of states, updated as the work runs and retries. A request is always in exactly one of them, and the application can read the current state at any time.

queued -> processing -> completed
queued -> processing -> retrying -> processing -> completed
queued -> processing -> failed

Submit a request

Install @reqrun/sdk and call it with the same chat completion shape you send to OpenAI. Supply the project key and signing secret once, when you construct the client; the SDK signs each request for you. Set wait to false so ReqRun accepts the request and returns an identifier without holding the connection open.

The example runs on Node 18 or later and needs two environment variables: the project API key and the signing secret shown once when the key is created.

import { ReqRun } from "@reqrun/sdk";

const reqrun = new ReqRun({
  apiKey: process.env.REQRUN_API_KEY!,            // reqrun_live_... or reqrun_test_...
  signingSecret: process.env.REQRUN_SIGNING_SECRET!, // reqrun_sign_... shown once at key creation
});

// wait=false: ReqRun accepts the work and returns a durable id without holding the connection open.
const accepted = await reqrun.chat.completions.create({
  model: "gpt-5-nano",
  messages: [{ role: "user", content: "Summarize incident 421." }],
  wait: false,
  idempotency_key: "incident-421-summary",
});

if (accepted.object === "chat.completion.async") {
  console.log(accepted.id, accepted.status); // rr_... queued
}

The response is a durable id

With wait set to false, the response is not the model's output. It is an acknowledgment: a durable rr_ identifier and a status. Store the identifier alongside your own record. This is the object that was missing whenever a request disappeared.

{
  "id": "rr_abc123",
  "object": "chat.completion.async",
  "status": "queued",
  "created": 1710000000
}

Check the status

When your application needs the result, it asks ReqRun. reqrun.requests.get(id) calls GET /v1/requests/{id} and returns the current status, the number of attempts, and the result once it exists. A hang becomes a pending status, and a dropped connection becomes a reason to check again rather than a dead end.

// Ask ReqRun for the outcome whenever your app needs it.
const request = await reqrun.requests.get(accepted.id);

console.log(request.status, request.attempts, request.last_error_code);

if (request.result) {
  console.log(request.result.choices[0]?.message.content);
}

The status response

The status response provides the resolved state the original connection could not. A completed request includes the result. A failed request includes a safe error code. Any state in between confirms that the work is still running in the service.

{
  "id": "rr_abc123",
  "status": "completed",
  "attempts": 2,
  "created_at": "2026-04-20T10:00:00Z",
  "updated_at": "2026-04-20T10:00:07Z",
  "last_error_code": null,
  "result": {
    "id": "chatcmpl_...",
    "object": "chat.completion"
  }
}

Why requests hang

A request hangs when the model takes longer than the client, function, proxy, or stream will wait. The call can still be running upstream after the connection that started it has closed.

The delay itself is a latency issue. It turns into a reliability issue because the request has no identity outside that one connection. When the connection ends, so does any way to learn what happened.

Confirming a request finished

The calling connection cannot tell you. A disconnect says nothing about whether the upstream work completed. You need a durable record that holds the outcome independently of the original request.

ReqRun maintains that record. Because it ran the request, it has the result regardless of what happened to your connection, and the answer becomes a status lookup: completed, failed, retrying, or still running.

When a longer timeout helps

A longer timeout can let a genuinely slow request finish before the deadline, but it does not resolve unknown state. It only postpones the point of uncertainty.

If the connection drops, the process restarts, or the user refreshes, the same question returns. Durable identity and status remove the uncertainty because they live in the service, not in the connection.

Summary

A request you cannot look up is a request that can disappear. The hang is the visible symptom; unknown state is the underlying cost.

ReqRun assigns every request an identifier when it is accepted and a status you can read at any time. An absent response no longer means an unknown outcome.