When an automated trading system loses its connection after sending an order, it may not know whether the order reached the broker. Retrying at once can place a second order. Doing nothing can leave the strategy unaware of a live position. The dangerous case is not a clear rejection; it is an uncertain result, when the request may have succeeded but the acknowledgement never arrived.
Preventing duplicate orders requires more than checking whether the system is connected. The trading application needs a durable record of each intended order, a controlled way to submit it, and a process for reconciling its local records with the broker’s order and execution records after a timeout or restart. This guide covers those controls, along with what to do when an API call, network connection, or broker service fails.
How duplicate orders happen
A common failure starts with an apparently simple sequence: the strategy generates an order, the application sends it, and the connection drops before the response returns. The order may have reached the broker and be working, even though the client received no acknowledgement. If the application treats the timeout as proof that the order failed, it may submit the same trade again. The account then has two orders where the strategy intended one.
The reverse can happen, too. The application records an order as submitted, but it failed before the request left the machine. If the system assumes the order is live and never checks, it may miss a valid trade. A crash between storing the order locally and sending it can create another ambiguous state. Network delivery and broker acceptance are separate events; a successful connection, a completed API request, and an accepted order are not interchangeable.
Common triggers include a client timeout, temporary loss of internet access, an API rate limit, a broker service interruption, an application restart, or a repeated strategy signal. A user clicking “retry” can cause the same problem as an automatic retry. The central design rule is straightforward: when submission status is uncertain, do not blindly send a fresh order. First establish what happened to the original request.
Give every intended trade a durable identity
Create an internal order intent before submitting anything. It should have a unique identifier and a record of the decision that produced it: the strategy or signal, account, instrument, side, quantity, order type, relevant price instructions, and creation time. Save the intent to durable storage before the network request. If the application crashes, it can recover the intent and check its state instead of generating another order from scratch.
Keep the identity of the strategy’s intent separate from the broker’s order identifier. The internal ID answers, “Which trading decision is this?” The broker ID identifies a request or order in the broker’s system. A client-supplied order ID can connect the two, but its behavior is defined by the broker’s API. Do not assume that reusing an ID makes a submission safe: some APIs reject duplicates, some return the original order, and some may treat repeated requests differently. Verify the documented behavior before relying on it.
Use a database constraint or equivalent atomic check to prevent two workers from claiming the same intent at once. This matters when a scheduler, a restarted process, and a second application instance can all act on the same signal. A “check, then write” operation without an atomic lock can let both workers see that no order exists and submit one each. Store the submitted request and its identifiers, too, so recovery can compare the original parameters with the current broker record.
Track order states explicitly
Do not represent an order with a single vague status such as “sent” or “failed.” Record distinct stages, for example: intent created, submission started, acknowledgement received, working, partially filled, filled, cancel requested, canceled, rejected, and status unknown. The exact names can vary, but the system must distinguish an order request from the order’s confirmed state at the broker.
After a timeout, mark the order as unknown or pending reconciliation, not rejected. That status tells the strategy to pause any replacement submission while the application checks the broker. An HTTP success response may mean only that an API request was processed; it does not necessarily mean the order was accepted or filled. Use the broker’s documented order status and execution updates as the authority for what happened.
Order protocols also distinguish a cancel request from a completed cancellation. The FIX Trading Community’s order and execution report specification notes that a pending cancel confirms receipt of the request, not that the order has been canceled. A cancellation must be confirmed; until then, the original order may still be active or may fill. The same caution applies to replacement requests. The article on order management, partial fills, and position reconciliation covers the wider lifecycle.
Make retries safe, not automatic
Retries are useful for temporary failures, but an order submission changes financial state. Repeating it can have a second effect. In software terms, a request is idempotent when repeating the same request produces the same result as performing it once. Some APIs support an idempotency key for this purpose; others do not. Even where a client order ID exists, confirm whether it provides deduplication for new orders, for how long, and what response a repeated request returns.
For a broker API that explicitly supports idempotent order submission, use the documented key consistently for retries of the same intent, with unchanged order parameters. A new trading decision should receive a new intent ID. Do not reuse an old key for a changed quantity or price unless the API explicitly defines that behavior. The general pattern is described in the AWS documentation on idempotent API requests; it illustrates why a repeated request needs a stable key and matching parameters. It is not evidence that a broker API supports the same feature.
If the broker does not document idempotent order submission, treat a timeout as ambiguous. Query by the client order ID if supported, check open orders and recent executions, and reconcile the account’s positions. If the broker has no reliable lookup by client ID, use the available order history and contact its support channel when records remain unclear. Do not submit a replacement simply because a status query returned no result once; the query itself may be delayed or incomplete.
For retryable errors, use a bounded retry policy with increasing delays, rather than sending requests continuously. Stop retrying after a defined limit, and route the order to reconciliation or a human alert. AWS’s retry and backoff guidance warns that repeated calls can add load and says retries should be used with idempotent operations. For trading, add an important constraint: an order that may already have been accepted should not be treated like a harmless read request.
Reconcile before resubmitting after a connection failure
When the API connection drops, pause new submissions for the affected account or instrument until the application has checked the unresolved orders. Reconnect using the broker’s documented process, then request current open orders, recent order history, fills, and positions as appropriate. Compare those records with the durable local intents. Check both the order’s status and its cumulative filled quantity; a partially filled order can leave exposure even if the remainder was canceled or rejected.
Reconciliation should be repeatable and safe. Fetching order status does not itself place another order, so a failed status query can usually be retried with care. A submission is different. If the broker cannot be queried, keep the intent in an unresolved state, stop the strategy from creating a duplicate, and alert an operator. The account may be exposed to price movement while the issue is unresolved, but guessing at the state can compound the problem.
Do not rely on positions alone. Two offsetting or partially filled orders can produce a position that looks plausible while open orders remain active. Conversely, a position update may arrive before a delayed order update. Reconciliation should compare orders, fills, and positions, and should account for timestamps or sequence information where the API provides them. A mismatch should create an exception for review, not be silently overwritten with whichever record arrived last.
Handle cancel and replace requests with care
A cancel request does not erase the original order. While cancellation is pending, the order may still fill. If the strategy submits a new order before the first is confirmed canceled, both can execute. A cancel rejection also leaves the original order in its prior state, so the application must read the response and update its local order record rather than assuming cancellation succeeded.
Replacement requests carry similar risk. If the system changes a price or quantity and then loses the connection, it needs to determine whether the original order remains active, the replacement was accepted, or a fill occurred during the change. Preserve the original order ID and parameters, record the replacement request separately, and wait for confirmation before treating the new terms as active. When the broker’s API supports status queries or event streams, use them to resolve the order rather than sending another replacement on a timer.
Partial fills require particular attention. A request to cancel the unfilled remainder does not reverse shares, contracts, or other units already executed. Update exposure from execution reports, not only from the intended order quantity. If the strategy decides to trade again, calculate the new quantity from the reconciled position and remaining live orders. The guide to connecting a trading system to a broker API covers the broader integration choices.
Prepare for restarts and lost event updates
Many systems receive order updates through a streaming connection or webhook. These updates can be delayed, duplicated, or missed while the client is disconnected. Design the application so that handling the same update twice does not count a fill twice, and so a restart can recover from the last saved state. Where available, use event IDs, sequence numbers, or execution IDs to detect repeats. Periodically compare the local view against broker records rather than trusting a stream as the only source of truth.
Persist updates before acknowledging or discarding them, and keep an audit trail that records the event, timestamp, associated intent, and resulting state change. If the data store is unavailable, do not keep submitting orders as though the system can safely track them. A transactional outbox pattern can help coordinate saving an order intent and scheduling its submission, though it does not itself prevent the broker from receiving a request twice. Broker-side idempotency or careful reconciliation is still needed.
On startup, load unresolved intents first. Fetch broker state, reconcile open orders and executions, then enable new order generation only when the account view is consistent enough to proceed. A restart should not replay every historical “submit” action. It should resume each intent from its recorded state and resolve uncertain submissions before taking new action.
Set limits and alerts around failure states
Build pre-trade checks that reject an order if the system already has an unresolved intent for the same strategy decision. Add limits for order size, total open quantity, position exposure, and submission rate. These checks should apply even when the model signal looks valid. A strategy should not be allowed to issue a burst of orders because a timeout caused its scheduler to repeat the same task.
Use alerts for repeated timeouts, rejected orders, reconciliation mismatches, a growing queue of unresolved intents, and unexpected order or fill counts. Log enough to reconstruct what happened: intent ID, client and broker order IDs, parameters, request and response times, retry count, connection state, order events, and resulting exposure. Keep sensitive credentials out of logs. For practical oversight, connect these controls to the system’s logs, alerts, and monitoring process.
A kill switch should prevent new submissions immediately, but it should not be mistaken for a universal cancel button. Cancel-all requests can fail or time out, and open orders may continue to execute until cancellation is confirmed. After activating a stop, reconcile the account and verify order status. Define which actions remain available during an incident, who can restart trading, and what conditions must be met before the system resumes. Related safeguards are covered in trading limits, kill switches, and human oversight.
Test failure scenarios before live trading
Test the awkward cases, not only the clean path where every request receives an immediate response. Simulate a timeout after the broker accepts an order, a disconnect before submission, duplicate signals, repeated API responses, a restart during a pending cancel, a partial fill followed by a connection loss, and a delayed status update. Confirm that the system does not send a second order unless reconciliation or documented idempotency establishes that the first request did not create one.
Run these tests in a sandbox or with recorded test data, then paper trade the full workflow. Simulated trading can show whether the application handles its own state transitions and recovery logic, but it cannot guarantee that a live broker connection will behave identically. Before enabling live order submission, set small exposure limits, test the stop procedure, and define a human escalation path. See the guide to what paper trading can establish about an AI system for the limits of simulated results.
In the United States, SEC Rule 15c3-5 imposes market access risk-control duties on broker-dealers that provide market access; the SEC staff notes that controls include preventing erroneous and duplicative securities orders. That rule is not a blanket technical specification for every retail trading application. It does underline why order checks and exposure controls matter, while the legal duties depend on the entity and activity involved. The SEC’s market access risk-control guidance describes the scope of those broker-dealer requirements.
A practical recovery sequence
-
Stop fresh submissions for the affected account, strategy, or instrument while the order state is uncertain.
-
Load the durable intent record and identify the exact client order ID, parameters, submission time, and last confirmed state.
-
Reconnect and query broker orders, executions, and positions. Retry read-only queries if needed, using sensible delays.
-
Match broker records to the intent. Account for partial fills, pending cancel or replace requests, and delayed updates.
-
Only after the original request is resolved, decide whether the strategy still calls for a new order. Give a genuinely new intent a new identity.
-
Record the resolution, alert an operator if records conflict, and resume only after configured exposure and system checks pass.
The safeguard that matters most is a clear distinction between “request timed out” and “order did not happen.” A trading system must treat uncertain submissions as unresolved, reconcile with the broker, and block duplicate actions until the state is known. That discipline turns a connection failure into a controlled interruption instead of an accidental second position.