Some bugs announce themselves with a stack trace. Others wait until a packet asks politely to be let through, and then say nothing at all. This note is about one of the second kind: a property of nftables that is documented, deliberate and entirely reasonable, and that turns one of the most natural rules you can write into a firewall that drops allowed traffic once a counter somewhere reaches its limit.
The firewall mentioned in the title was not eaten in production. It was eaten on paper during a design review. This is the cheapest place to lose a firewall. I am writing this because the rule that caused the issue looks correct. It reads correctly and passes every test performed on a quiet network.
The setting
My project manages a network gateway on a vessel. This gateway divides the ship’s networks into specific zones, such as the bridge, engine room, office, and crew areas. It also enforces communication rules between these zones. Before applying a new security policy, the gateway must gather traffic data. It tracks which flows each zone accepts or denies. It also identifies what would be blocked under the proposed policy. This evidence helps operators detect hidden traffic patterns, such as an engine room monitoring system communicating with an undocumented server, before they tighten any rules.
We collected this evidence in the kernel using nftables dynamic sets. A dynamic set allows rules to address data from the packet path. When a rule matches a packet, it builds a key from the packet’s fields. These fields include the source, destination, protocol, port, and verdict. The rule then adds this key to the set. Every fifteen minutes, the gateway reads these sets and creates summaries. Other methods, such as conntrack events or logging every packet to a user-space daemon, are less efficient. They require more processes, more moving parts, and create a higher risk of system lag.
A gateway on a ship has bounded memory, so every set is bounded too:
set flows-office {
type ipv4_addr . ipv4_addr . inet_proto . inet_service . mark
flags dynamic, timeout
timeout 1h
size 65536
counter
}
The system stores a maximum of 65,536 flows. Each flow is deleted one hour after it is added. The key question is what happens when flow 65,537 arrives.
How nftables evaluates a rule
An nftables rule is not just one condition and an action. It is a sequence of expressions evaluated from left to right. These expressions include payload loads, comparisons, set lookups, counters, set updates, and sometimes a final verdict like “accept.” Each expression can end a rule early. If a comparison fails, the expression sets an internal “NFT_BREAK” verdict. The chain loop in “nf_tables_core.c” interprets this as a signal to skip the current rule and move to the next one. This process explains how matching works: a rule is skipped if any part of it fails early.
The key detail is that comparisons are not the only expressions that break. The dynamic set expression, ‘nft_dynset’, writes ‘NFT_BREAK’ if it cannot add an element. This happens, for example, if the set has already reached its size limit. Similarly, the rate limiter, ‘nft_limit’, triggers this if a packet exceeds its rate. To the chain loop, a failed add and a failed match are identical. Both result in skipping the rest of the rule.
The kernel views this as consistent and sensible. An “add” is an expression like any other. If an expression fails, it should stop the rule from continuing. For the person writing the ruleset, the word “add” implies “only if there was room.”
The rule that eats the firewall
Everyone writes this rule first. It records an accepted flow and accepts it in a single line. This approach is tidy and reads exactly as intended.
ip saddr 10.10.10.0/24 tcp dport 443 \
add @flows-office { ip saddr . ip daddr . meta l4proto . th dport . 0x1 } \
accept
The set functions correctly as long as it has room. When the set becomes full, the ‘add’ operation fails. The ‘accept’ action is not reached, so the packet moves to the next rule. In a zone chain, subsequent rules belong to different zones and do not match the packet. The process eventually falls through to the chain’s policy, which is set to ‘drop’. As a result, allowed traffic is denied by a rule that is labeled ‘accept’.
Several features of this failure make it worse than an ordinary misconfiguration.
- The process is silent. It returns no errors and logs nothing. The ‘nft list ruleset’ command displays the ruleset exactly as it was loaded. If the rule has a counter before the ‘add’ command, that counter continues to increase.
- This process is selective. If a key already exists in the set, the system updates it rather than adding a new entry. This operation succeeds. Most rulesets accept established connections first, allowing them to continue. Only new flows will fail. Furthermore, this applies only to flows with keys not yet in the set.
- It heals itself. Elements expire after one hour. A room appears, and the rule resets. This continues until the set fills up again.
- Load triggers this issue, not configuration. A misbehaving device, a network scan, or a laptop syncing photos over many short connections can fill the set. Typically, the device that loses connectivity is not the one causing the issue.
Field reports often state: “The chart server intermittently fails to connect. It works again an hour later, and the firewall shows no issues.” This typically prompts the response, “Have you tried restarting it?” In many cases, a restart appears to resolve the problem.
The fix, part one: a rule shape
A better fix is not a cleverer rule. Instead, impose a constraint on all rules: an ‘add’ action must never share a rule with a verdict. Recording and deciding should be separate rules that share the same match. If the recording rule fails, only the record is lost. The packet then continues to the next rule to receive its verdict.
In the rendered ruleset each recorded flow becomes three rules:
# 1. A flow the set already holds: a plain lookup, no add, no verdict.
<match> <key> @flows-office
# 2. A new flow: framed by two counters, added under a per-source share, no verdict.
<match> <key> != @flows-office \
counter name "seen-flows-office" \
add @gate-flows-office { ip saddr limit rate 1/second burst 512 packets } \
add @flows-office { <key> } \
counter name "kept-flows-office"
# 3. The verdict, in a rule of its own, with the same match.
<match> accept
The second rule now breaks for two intentional reasons. First, the add breaks if the set is full. Second, if a source device adds more than 512 flows at a rate exceeding one per second, the rate limiter triggers. When this happens, the system discards the device’s flows entirely. This mechanism enforces a per-source share. It prevents a single device from filling the set and hiding the flows of others. Ironically, the very mechanism that caused the original problem is now what enforces this share. ‘NFT_BREAK’ was never the enemy; the true issue was sharing a rule with it.
The same discipline applies to chains that evaluate a candidate policy alongside the enforcing one. These chains count what the candidate would allow or deny and record its denials. They do not contain ‘accept’, ‘drop’, or jumps into enforcing chains. Therefore, a packet passes through them as if they were not there, regardless of whether the sets are full.
The fix, part two: measuring what was not recorded
Separating the add from the verdict keeps traffic flowing. However, this process does not signal that evidence is missing. Missing evidence is a problem in itself. A summary based on complete data might claim certain flows never occurred, even though they were simply never recorded.
The two counters in the recording rule explain this. ‘seen-<set>‘ sits before the add operation. It counts every packet of a new flow. ‘kept-<set>‘ sits after the add operation. It only counts packets where the add and share succeeded. If an expression fails, a break ends the rule. Everything after that point is skipped, including the second counter. The overflow is the difference between these two values.
overflow(set) = seen(set) - kept(set)
This costs two counters per set. There are no extra rules. It works the same regardless of why the break occurred. It requires no cooperation from the kernel beyond the initial problem-causing behavior. The counters are named objects in the table. Therefore, they remain when a new policy replaces the ruleset.
On 2026-10-24, a lab test was conducted using Debian 13 and kernel 6.12. A flood of 5,083 new flows entered a zone from test devices. Of these, 4,070 flows were kept and 1,013 overflowed. Forwarding remained unaffected throughout the test.
A measured overflow then becomes an input rather than a footnote. In my project, a zone whose flow sets overflow raises an alert and holds any pending policy change for that zone while the overflow lasts and for 24 hours after, because the evidence on which that change would be judged is incomplete. An operator can release the hold; the system does not release it on their behalf.
The fix, part three: testing the shape, not an example
A compiler generates the ruleset from a policy. It uses golden-file tests for validation. These tests render a policy and compare the output against a pre-approved file. While golden files are useful, they only detect if the output has changed. They do not confirm if the output is correct. A developer might approve a file containing simple rules while they are distracted. Consequently, these tests may not catch significant errors.
There is a second test. It reads like a property rather than an example. It renders several policies with different shapes. These include a reference vessel, a zone with a staged candidate, and zones with crew networks. Then, for every line that adds to a flow set:
- the rule contains no verdict: no `accept`, `drop`, `reject`, `return`, `jump`, `goto` or `queue`;
- the line before it is the plain lookup of the same key;
- it begins with the `seen` counter, adds under the source's share, and ends with the `kept` counter;
- the key it adds is exactly the key it looked up;
- a verdict follows later in the chain, in a rule whose match is the match the record used.
The test ensures each policy contains more than ten lines. This prevents a change that stops recording flows from passing the test accidentally. If a future compiler update places an ‘add’ next to an ‘accept’, the build will fail. This remains true regardless of the zone, key form, or policy shape, even if the original reason for the rule is forgotten.
What remains
Three costs are worth stating.
- The ruleset is now longer. Each recorded flow consists of three rules instead of one. Additionally, the system evaluates the match more than once. We have not yet measured the performance cost. It is not free, and on a high-rate link, this cost must be measured rather than assumed to be negligible.
- The “per-source” share applies to each source address. A device can forge source addresses to claim multiple shares. Using 128 forged addresses with 512 flows each can fill a 65,536-element set. Once the set is full, the system ignores new evidence for up to an hour. This delay can postpone a pending policy change. However, the device cannot change which traffic the firewall forwards or drops. Protecting that forwarding behavior is the main purpose of the rule shape.
- The counters measure packets from unrecorded flows, not the flows themselves. This shows that the evidence is incomplete and estimates the extent of the loss. However, it does not provide enough information to reconstruct the missing data.
Lessons
This lesson predates nftables. It applies when a required operation follows an operation that might fail. If the first operation fails, the second cannot proceed. In programming, developers notice this because functions return values. Firewall rules do not return values. Instead, success depends on whether the packet reaches the end of the rule set.
Three practical rules apply to any ruleset that writes to sets from the packet path:
- First, keep stateful side effects out of rules that carry a verdict, unless the verdict depends on them. These effects include additions, updates, meters, and rate limits.
- Second, if a side effect can fail, use counters to track it. This ensures that failures appear as numbers rather than absences.
- Third, test the invariant against everything the generator can produce. Do not rely on a single approved example of its output.
The next note in this series will explore another situation where a reasonable default met an unreasonable environment.
References
- Linux kernel, `net/netfilter/nf_tables_core.c`: rule evaluation loop and the handling of `NFT_BREAK`.
- Linux kernel, `net/netfilter/nft_dynset.c`: the dynamic set expression, which breaks the rule when an element cannot be added.
- Linux kernel, `net/netfilter/nft_limit.c`: the rate limit expression, which breaks the rule when a packet is over its rate.
- nftables wiki, "Sets" and "Updating sets from the packet path", https://wiki.nftables.org/.
