Two operational problems come up on almost every NiFi estate we look at. Queues back up and someone raises the threshold. The provenance repository grows until a disk fills and someone shortens retention in a hurry. Both have better answers.
What back-pressure is for
Every connection in NiFi has a back-pressure threshold, by object count and by data size. When a queue reaches either limit, NiFi stops scheduling the component that feeds it. The defaults on a new connection are 10,000 FlowFiles or 1 GB. The idea is simple: a slow consumer should slow the producer down rather than let data pile up until a repository disk is full.
Back-pressure is a signal. When a queue is at its threshold, something downstream is slower than something upstream.
Read the queue before you change it
When a queue backs up, ask three questions before touching the threshold.
Is the consumer stopped, failing or slow? A stopped processor, one bouncing FlowFiles to a failure relationship that loops back, or one with a growing number of bulletins is not a capacity problem. Fix the cause.
Is the downstream system the bottleneck? A processor writing to a database, an API or a file share can only go as fast as that system accepts. More threads may simply mean more concurrent requests against a system that is already saturated.
Is the problem the shape of the data? Flows that split large files into thousands of tiny FlowFiles produce queues with huge object counts and small sizes. Record-oriented processing, which handles many records inside one FlowFile, often removes the backlog instead of moving it.
Only when the consumer is healthy, the downstream system has headroom, and the data shape is sensible does more concurrency or a higher threshold make sense.
Set thresholds deliberately
The defaults are a starting point, not a design. A few principles:
- Size thresholds to the content repository. The sum of what the queues can hold should fit within the content repository disk with room to spare. Raising every threshold to a huge number just moves the failure from a full queue to a full disk.
- Use the size limit for large payloads. A queue of large files will hit the size limit long before the object count. That is usually what you want.
- Keep thresholds low in front of fragile systems. A small threshold in front of a slow API applies pressure early, which is easier to recover from than a large backlog.
- Watch swapping. When a queue holds more FlowFiles than the swap threshold, NiFi swaps their attributes out of memory to disk. That protects the heap, but a cluster that is constantly swapping is telling you the queues are too deep.
Size provenance to what you query
The provenance repository records what happened to each FlowFile: received, routed, modified, sent, dropped. It is invaluable for answering "where did this record go?" and it grows with traffic. Retention is bounded by time and by storage size, and whichever limit is hit first wins.
Start with the question: how far back do people actually look? On many estates the honest answer is a few days, because anything older is answered from the source or destination systems. Some regulated environments need longer, and then the right move is usually to export provenance events to a log or analytics platform with the retention you need, rather than keeping months on the nodes themselves.
Then size the repository to that retention with headroom, on its own disk, separate from the content and FlowFile repositories, so a burst of provenance does not starve the flow. Watch indexing: indexing many attributes makes searches convenient and makes the repository larger and slower to write. Index the fields people search on, not everything.
Monitor the signals that matter
Wire a few metrics into the monitoring your team already uses: queue depth and back-pressure state on the connections that matter, repository disk usage for content, FlowFile and provenance, JVM heap and garbage collection, and bulletin counts. Alert on trends, such as a queue that has been growing for an hour, rather than on single spikes.
Write it down
Whatever thresholds and retention you choose, record why. The next engineer who sees a queue at its limit should be able to read whether that limit was a decision or a default.
This kind of tuning is a regular part of our ongoing flow operations work.