Fast-forward to 2026: zooming in from the loo is probably acceptable as long as you remember to set the right beach background and your app version and the on-device audio ML stack can filter out the plop. We are technologically native and fluent at mains and alts and soft-launches and Partifuls.

Yet I can not seem to stop arguing with my LLM. It is an immutable set of weights that talks like a human and even mocks offense like a human. But it does not live and die on hills like a human. It does not have any notion of dignity where when consistently proven wrong, it reflects, reassesses, and adjusts its approach. It has no sense of how many tokens are necessary to service the conversation—they are churned out using industrial-scale production facilities, trivially saturating my cognitive capacity to filter slop. At its worst, it deploys my subscription fees into consuming my time and cognitive bandwidth. I know this but I can not stop—it riles me up like a wrong human. My object has moved from a stranger on the internet with a modicum of life to an on-demand stranger-aping function on my phone.

Maybe this is a me problem. Maybe the primary content curation vehicle for our society has not become uninterpretable algorithms that automatically learn what makes us engage. Maybe it is not the case that this creates an adversarial relationship between the platform and the user, where industrial-scale tech is deployed not to service the user, but to hijack them into inaction. Maybe it is not the case that the social is strictly being substituted by the parasocial. At least the cocaine users in the 80s hung out with their friends. At least the DEA hunted Pablo Escobar.

Before I continue further down this path: let me draw a hard line. No concession is intended to Luddites. We must do newer and greater things. The arc of technology must always trend forward. A good devil’s advocate argument that does not reject forward progress entirely is rarer than an inoculated user of technology, and this post is intended to fill that gap.

Now the specific problems with say reels are a point of this post but not the point. We will suffer and then inoculate ourselves against reels, eventually. Fast-forward a hundred years, maybe your attention and dopamine are monitored by your PCP. Maybe Instagram gets an equivalent of a nutrient label or a broadband facts label. It is no different from polluting our air and then legislating clean air and water and environmental protection acts. It is not a guarantee as much as a survival bias assumption: it will either happen or we perish. And humanity is absolutely underrated and absolutely goated at not perishing, but the opposite of that at not letting problems fester and attacking them head-on well before our backs touch a wall.

What is the point? It is 1) to recognize this phenomena, and 2) liken it to novel pathogens that we eventually develop antibodies against, and 3) reflect on whether our current infra for doing so is appropriate. It is a cop-out to invoke a government here. Governments are formalizations of society’s instincts for order, but are increasingly perceived as excuses to justify atrophying of the aforementioned instincts. Without bothering to unpack this further, I will state that the continued operative presence of the instinct is a necessary precondition for Liberalism to endure. Maybe this is just an example of how The Search works at the frontier. Maybe adapting to pathogens is just backtracking along the constituent axis and consolidating resources for forward progress along a different one. A satisfying conclusion to this post would require a satisfying conclusion to the problem, and I do not have one yet.

1. Scaling debates in filesystem metadata, predominantly in HPC where we like to have 100k processes synchronously hammer our shared Lustre. Relatively outrageous solution proposals such as no-global-namespace¹, along with more conventional userspace and overlay filesystems.
2. The rise of object stores² and debates on whether we need a filesystem at all³.
3. The rise of Parquet, open table formats like Iceberg⁴ and Delta Lake⁵, and things like Liquid Clustering⁶.
4. Ankush discovering early in grad school that file formats with attributes as contiguous buffers are great for particle analytics, manifest formats describing a set of these files along with some per-file statistics, and opportunistic partial ordering run circles around almost everything⁷.
5. Ankush discovering that telemetry persisted as JSON or CSV is one of the leading causes for global suffering among those trying to understand and fix their systems⁸, and that things go much faster once you use this Pandas format called feather.
6. Ankush finally embracing the daylight and building a system for trace-event analytics around in-situ dataframe analytics and timestep-partitioned Parquet. Missing a manifest format that would just bundle everything together, and learning about Iceberg/Delta Lake (DL) as the paper is being wrapped up⁹.

I will now stop referring to myself in third-person. Ok, we get it: columnar analytics is great. For enterprise data, particle data, even trace data. But this is 2026, we know this, and it is not exactly the second coming of Yahoo Messenger. But this becomes the backdrop against which we evaluate how a filesystem and metadata should and can look like.

Before I go deeper into metadata, let me emphasize that columnar analytics seems to me to be more general than you might think. The world of scientific computing is packed with oddball use-cases that OLAP would not have encountered: N-D arrays, HDF5, irregular meshes etc being some of them. But it turns out that HDF5 is just contiguous buffers and pointers. Your data structure might be 3D but your memory is not, and we really only get to preserve one dimension. Do we really need multiple container formats for contiguous buffers? How far does the dominion of columnar analytics formats really extend? A frontier, if one exists, is beyond my line of sight.

With that, let us now get to metadata. In the face of the rising tide of object stores, I have argued that you cannot just have your storage system be a zillion objects. One day you are going to ask: who do these zillion objects belong to and can I delete them? An instructive lesson from politics is that when people act irrationally, it is not sufficient to critique irrational behavior—however gratifying it might be. The rational side must reflect on why it lost some customers. This is obviously directed at POSIX.

When 100,000 ranks rise in harmony and hammer their filesystems with bits, they are not publishing relevant facts in the moment. It is only when the storage phase, orchestrated by a lead rank, wraps up that a publishable entity emerges—an analytical dataset. It is fruitless to create durable metadata from 100,000 ranks synchronously. The objects must be allowed to be written without filesystem interference and then committed atomically via a manifest file (or hierarchies thereof) by the lead rank. Hierarchical in-fabric aggregation is how anything scales in a supercomputer. We just never extended this principle to storage. The commit artifact—the manifest—not only tells you about the existence and the location of its constituent objects, but it also contains indices and statistics that can enable query planning and optimization and pruning. Now this is metadata, and POSIX never stood a chance.

The other oddity database folks conjured up and I just learned about is a Metastore. To me, it seems to be their take on a filesystem, with catalogs and schemas and tables. Now there are plenty of SQL tables in the world and a namespace to manage them seems natural, and if metastores were everything, DBFS would not be a thing… but it does play a role in the data management lifecycle in the Iceberg world by facilitating atomic commits of Iceberg/DL updates. HPC does not really need metastores, I suppose, if you do not require your Iceberg/DL metadata to span filesystem boundaries you can achieve the same outcome with a single Lustre instance.

One must give this to databases: they evolved principled definitions of consistency and isolation semantics, and you inherit these properties if you build your namespaces on top of these tools. Metastores share these characteristics, and as a result you get a richer description of your world that allows more applications to do more things. But the road to this heaven is gatekept by Codd and schemas and constraints are the toll one must pay to enter. For when they commit the sin of bearing digital contraband of uncertain structure and provenance, heathens will always find refuge in a filesystem.

My pitch for HPC, to be clear, is checkpointing as a commit protocol, with ranks writing portions of a dataset independently, and generation of a manifest describing these portions orchestrated by a multi-tier reduction hierarchy. The root of this reduction hierarchy commits the epoch to the filesystem after all objects are in. These catalog and container formats are common and interoperable, with former not only enumerating/locating the latter, but also indexing statistics for analytics. In addition to facilitating the commit, the reduction hierarchy also enables streaming reorganization and in-situ analytics/filtering/aggregation before any bit hits a durable layer. Take-home question: what does a scientific catalog look like?

Aside: MapReduce > SQL. I suppose there was no reason to go down this route, but reason is not why this blog exists. Some flavor of MapReduce seems to me to the best version of general-purpose analytics, and also more applicable to scientific analytics than we might think. MapReduce is the best runtime for OLAP because it is more general than SQL, but not terribly more complex. ChatGPT says that Stonebraker said something around this and that it might be wrong. I do not trust ChatGPT. Anyway, beyond a shuffler, all SQL operators are just different flavors of map functions that you can reason about a bit for optimization. Note that MapReduce does not necessarily mean Spark: the multi-tier columnar reduction and in-situ analysis framework described above is coming with [9].

References

Let us first explore the origin of prediction, and on that quest, let us first try definitions of the term and see what fits.

• Is it… using the rules of the environment to tell what happens tomorrow? A ball dropped from a 10m building will hit the ground in 1s (numerical accuracy of the basic physics employed here is not necessary for this exercise). Is this just science?
• But what if the rules are intractable? There exist methods for predicting weather, but let us say I do not have access to this weather prediction infra. I can make conditional predictions: if there are clouds, it will rain.
• Will it definitely rain if there are clouds? Here we branch into science and bayesian reasoning. Science allows definite claims with errors… or does it? Weather prediction is also probabilistic, with ensemble models and what not. So is science just “principled application of hand-wave” and bayesian reasoning “vibes-based hand-wave”?
• What is bayesian reasoning? This one is easy: it is just the application of formal principles to a set of probabilistic inputs. It does not concern itself with how they were derived but just allows principled synthesis, and as such is not really a primitive.
• Is the environment always deterministic, i.e. does it always lend itself to a predictive model? One source of uncertainty is rapidly evolving fine-grained phenomena that we refuse to model. Air molecules have frenetic and unique trajectories, but for most purposes it suffices to model air as having a temperature: statistical physics where we can model an ensemble without needing to model individual entities, insofar as the behavior is amenable to this approach. Another source of uncertainty is agentic behavior: what will the drivers and pedestrians do on the road tomorrow?
• Is inferring the rules of the environment necessary for prediction? ML models predict all the time without learning natural laws, while humans supposedly do learn natural laws. The natural law process seems to be highly robust and while its origin is hard to point out, millions of years of evolution enable a circular description: principles that reliably confer survival.

Ultimately, the goal of prediction is the agent’s survival. Famines mean you ration food. Rain means you pack an umbrella. Reasoning is an approach to prediction that does not accumulate error, and hence be compounded and composed indefinitely. Note that this does not salvage reasoning, because the process of modeling reality into reasoning frameworks is intrinsically error prone, and it is trivial to arrive at formulations of problems that are not amenable to zero-error solutions. We therefore try relaxations where we deliberately and consciously accept errors for approximate solutions.

But regardless of whether it is math or ML models, prediction has a finite horizon and humans deal with it all the time. The model said it would rain but did not. Claude said it fixed the bug but it did not. The key to human survival is not the NWS or Sam Altman, but the ability to respond to the residual between reality and prediction. Note that unless we can perfectly simulate the universe, a non-zero residual exists by definition, and the only tool that tames that monster is adaptation.

In fact—and we finally arrive at the core thesis of this post—adaptation is the necessary and sufficient condition for agents’ existence, and prediction is just a special case of approaches for reducing adaptation latency. If you can anticipate, you can formulate strategies that you could not have otherwise. But it is the capability that picks up the tab when prediction falls short, which it does by definition.

So what is the point? LLMs are predictive engines. They are a statistical compression of our knowledge, like a generative textbook. Just like it is a category error to say my math textbook is smarter than me or can do my job, the same applies for LLMs. This is not to say that there can not exist a pytorch-executable DNN that can adapt. The argument is simply that we have no evidence of that capability emerging as a result of all the scaling over the last few years.

Why this might be good, actually. An example of adaptive systems is the bubonic plague. Another example is the COVID-19 virus. These and multiple examples in their class demonstrate that the upper bound for an adaptive system is devastating. These systems are always fuzzing against their environment to replicate. Note that intent is a poor rubric to reason about agents: the COVID-19 virus did not wake up and decide to be mean to me. That predictive systems can not grow adaptive bones is good. It makes it impossible for a Waymo to one day emerge the ability to decide that it does not my incendiary statements against machines and run over me.

What does this mean for jobs? A piece of technology does not need to be AGI to profoundly affect jobs. Cars affected jobs, and so did washing machines, batteries, computers, smart phones etc. There is a transition period where accountants argue that the tech will eliminate their jobs, a new equilibrium of higher productivity is reached, peace holds for a few years until the next such thing. This one is no different.

For a positive conclusion (positive as in predictive, not ruling-out), let us apply Amdahl’s Law to jobs: currently they are $x\%$ prediction and $y\%$ adaptation. Take coding: sometimes I need to stare at core dumps, other times I am writing printf statements to assist with debugging. The parting thought is this: what happens if we apply infinite speedup to the $x\%$?

EDIT: Actually, I lied. Modern LLMs are somewhat recursive: they predict, there is a residual, they then learn to predict the residual, and so on, and the amount of time they can work before the leash needs to be yanked goes up. The residual framing tells us that there must be an upper bound. But in the short run, the benchmark is not AGI. The benchmark is: can we find other avenues to divert human adaptivity to faster than LLMs gobble up the predictive components of their current avenue? The answer to this could very well no, and while you could argue for this to just be a system that has not achieved its steady state, systems’ unsteadiness can last longer than human careers.

Edit: I prompt-engineered some LLMs into generating this. Same reliability caveats as the rest of the post apply.

Losslessless: The Bedrock of RDMA

RDMA originated on a natively lossless fabric (Infiniband). Since then, we have tried to graft it into inherently lossy fabrics, with brittle outcomes. We like losslessness for the same reason we like strong consistency: it is nice and simple if things just work and are predictable.

1. Every network becomes lossless at low utilization.
2. CBFC is the best way we know to build a lossless network at higher traffic volumes. PFC is very fragile, intrinsically so.
3. CBFC and PFC represent two fundamentally differing approaches to utilization management: proactive and reactive, with their own tradeoffs. The cost of a proactive approach is “imposing flow control when it was unnecessary”, leading to potential underutilization.
4. There is a fundamental tradeoff between network utilization and loss guarantees. For any given lossless network, you can afford to be a little speculative and increase median usage. The null hypothesis is that this increases loss probability. Unless reasoned all the way through and demonstrated to not exist or not be a factor, it must exist.
5. We like lossless networks because in certain conditions, the cost of loss to higher layers is prohibitive. We should not design lossless schemes that dispatch speculatively without reasoning about why we wanted a lossless scheme in the first place, and why that speculative dispatch does not create problems we wanted to avoid¹.

Everyone Likes Most of Infiniband

Let us disaggregate infiniband into a series of features.

Likes:

1. Everyone likes CBFC[^1,2,3]
2. Everyone likes the performance benefits of RDMA, if not the complexity of memory registration and coding verbs.
3. We like the cache-friendy access patterns enabled by verbs as an interface. The TCP/IP stack, as implemented, has been noted to force cache-unfriendly behavior, creating utilization bubbles in the precious PCIe link². verbs, used along with capabilities such as SRQs, multi-receive, and hugepages enable intrinsically contiguous accesses.

Dislikes:

1. Everyone dislikes its difficult relationship with the broader datacenter network. It is not a part of the broader network, but a separate sub-network like NVLink or CXL.
2. Many folks dislike its scalability constraints (connections need QPs need NIC SRAM).

What We Really Need: RD + Ethernet Compatibility

Case Study: QLogic

QLogic already had this insight 15 years ago. QLogic is not super well known, and I think it is unfortunate. Many aspects of their approach are highly relevant to some debates we have had since.

QLogic Truescale was the precursor to Intel Omnipath, which was the precursor to Cornelis Networks. Instead of offload to a smart NIC, they wanted onload: CPU-mediated communication for scalability. It was a verbs-compliant fabric that natively spoke a different interface called psm.

Truescale was a messaging-oriented fabric. I think of it as RDMA UD with large datagram sizes. It injected small payloads inline into a packet, and large payloads were DMA’ed on both the send and receive paths using transient connection mechanisms called TID/TIDflows.

From what I understand, PSM had CBFC but no link-level retry (LLR). It had a software-based go-back-N-style recovery. I think link-level retry requires a smarter HCA, which they were trying to avoid, at least for the first generation. I have observed software-based retries in that fabric to add an extraordinary amount of jitter, when triggered, but it is triggered by some residual bugs and high timeouts, and not by packet drops.

Case Study: SRD in Amazon EFA

The variable-sized datagram design point remains compelling. Amazon built a fabric called Elastic Fabric Adapter (EFA) that presumably adds LLR to UD. I don’t know much about it so I’ll keep it short.

RDMA vs RPCs: A False Dichotomy?

I’m probably wading into a decade-old debate at this point, but it’s not my fault that I was just a swaddling infant a decade ago.

Let us take eRPC³, an iconic paper at this point. The argument I will explore is whether eRPC implements Infiniband in software over lossy ethernet, and whether while a solid step forward for lossy ethernet, its design just strengthens the argument for real RDMA/infiniband.

1. eRPC likes losslessness. They create those conditions by having session credits and limiting dispatched traffic to network BDP. Session credits resemble an end-to-end version of CBFC, something also tried in ⁴.
2. eRPC likes contiguous message buffers, an event loop for short messages, and memcpy’ed contiguous buffers for large messages, with memcpy orchestrated by CPU. This feels very close to the workflow I employ in my current codebase using the RDMA-capable Mercury RPC library, except using CPU cycles instead of RDMA.

I wonder what is the difference between “we found a lossless regime in the network and designed a system optimized for a lossless common-case” and “we engineered a lossless regime in the network, and this proves the value of a genuinely lossless fabric.”

I also wonder about the value of pushing retries to the application layer when the ultimate behavior you want is lossless. There are plenty of applications that tolerate lossiness end-to-end, and it makes sense for the lowest common denominator to not include retries for general-purpose networks. But datacenters are a controlled high-density environment, and maybe it is easier to permit lossy flows as a second-class citizen in a lossless fabric than the other way round?

Also, I think scalability criticisms of infiniband are hardly the killer argument they are made out to be. Truescale is an example of a production fabric that already solved those challenges, and software-level solutions such as RxD exist in libfabric (multiplexing RDMA over UD). RDMA is not infiniband, and there are multiple CBFC-based approaches to lossless fabrics.

Next, I think RDMA doesn’t preclude the ability to do two-sided communication. I think it is not completely accurate to think of RDMA as disaggregated memory. That is, RDMA is not CXL. An RDMA-capable node is essentially still a request/response-based server. Maybe it is useful to mentally decouple operations into metadata and data, and use RDMA for the data path (common criticism is unnecessary RTTs for pointer chasing). Mercury⁵, with providers for libfabric etc, provides both messaging-style semantics for small messages and verbs-enabled RDMA via a bulk interface for large messages.

Finally, I think the end-to-end argument does not preclude merging of layers for performance. I think it is a neat guiding principle for what to do in the absence of competing concerns, but it does not preemptively override the presence of competing concerns.

What’s Next?

I want to understand HPE Slingshot and UltraEthernet better, and maybe also start dabbling with CXL.

Side note: do you know PCIe uses CBFC? I think the gap between an interconnect, a fabric, and a network is smaller than we think!

References

Updated on 20251008 and later on 20251014 to present a version that may age better.

Introduction

RDMA-CM (RDMA Connection Manager) is the control-plane layer for RDMA. It exists to make connection setup look and feel like sockets: applications can bind to addresses, listen on ports, and accept or connect without worrying about raw InfiniBand identifiers. Underneath, RDMA-CM exchanges the connection metadata, programs the QPs, and hands back an endpoint ready for verbs. This is distinct from the verbs API, which only covers the data plane.

Background: QoS in InfiniBand

InfiniBand provides Quality of Service using Service Levels (SLs). An SL is a small integer carried in each packet’s header. The subnet manager maps SLs onto Virtual Lanes (VLs), which are independent link-level flows with separate buffering and flow control. By assigning flows to different SLs, the fabric can prioritize some traffic or keep classes of traffic from interfering with each other. SL is selected when a QP is created, and remains fixed for that QP’s traffic.

RDMA-CM Basics

RDMA-CM exposes a socket-like API:

• The central handle is an rdma_cm_id, analogous to a socket fd.

• Each cm_id belongs to a port space, such as RDMA_PS_TCP (connection-oriented) or RDMA_PS_UDP (datagrams).

• Applications use rdma_bind_addr, rdma_listen, rdma_connect, etc., in the same way they would with sockets.

What looks like an IP+port pair in RDMA-CM is just a wrapper. With IPoIB enabled, the “IP” is the interface address (e.g. 10.94.3.105), which RDMA-CM resolves to a GID. Without IPoIB, you still can create connections, but the address is directly a GID (usually expressed in IPv6 link-local form). In both cases, the binding resolves to a specific HCA port and GID.

The handshake itself is a control-plane exchange: on InfiniBand it uses CM MADs over QP1, on RoCE/iWARP it uses UDP/TCP messages. This is how QP numbers, PSNs, MTU, and SL hints are exchanged.

RDMA-CM and Service IDs

The key to QoS integration is that RDMA-CM maps the familiar notion of port numbers into InfiniBand Service IDs. A Service ID is a 64‑bit identifier that the Subnet Manager understands. In RDMA_PS_TCP, the port number you bind is encoded into the Service ID automatically. This provides a clean classification hook: flows created on different port ranges correspond to different Service IDs.

Subnet Manager QoS Policies & Setup (OpenSM)

OpenSM can enforce QoS by mapping Service IDs (derived from RDMA-CM port numbers) to Service Levels (SLs) and then to Virtual Lanes (VLs).

1) Enable QoS in OpenSM

Edit /etc/opensm/opensm.conf (or the file you pass with -f) and set:

qos TRUE
qos_max_vls 8              # adjust to your ASIC/link support (e.g., 4 or 8)
qos_policy_file /etc/opensm/qos-policy.conf

You may additionally pin SL→VL and VL arbitration globally here if you want static tables in the options file (advanced fabrics often keep these in the policy file instead).

Restart or start OpenSM with QoS active:

systemctl restart opensm    # distro services
# or manually
opensm -Q -f /etc/opensm/opensm.conf
# (-Q/--qos is equivalent to qos TRUE; -Y sets an explicit policy file path)

2) Define QoS policy (port→ServiceID→SL)

Create /etc/opensm/qos-policy.conf with policy rules. A pragmatic pattern is to carve the RDMA-CM port space into classes and assign SLs per Service ID range. Example with three classes:

# --- Classes ---------------------------------------------------
# Control plane / RPC (low bandwidth, low latency)
service_id 0x0000000000001000-0x0000000000001fff sl 4

# Bulk telemetry / background
service_id 0x0000000000002000-0x0000000000002fff sl 1

# Data path / latency-critical (highest priority)
service_id 0x0000000000003000-0x0000000000003fff sl 6

Notes:

• RDMA-CM encodes RDMA_PS_TCP + port into a 64-bit Service ID. Use contiguous Service ID ranges to represent your port ranges. Confirm the exact mapping in your stack by printing the CM service_id on connection or using SA queries (see verification below).

• If you use IPoIB, classification still hinges on Service ID, not L3 QoS.

3) Map SLs to VLs (isolation / priority)

Still in qos-policy.conf, define SL→VL and VL arbitration so the fabric actually separates traffic:

# Map each SL to a VL. Keep hot classes on distinct VLs.
sl2vl 0 0
sl2vl 1 1
sl2vl 4 2
sl2vl 6 3

# Basic VL arbitration (weights). Keep highest-priority VLs with more credits.
vlarb_high 0:64 1:64 2:96 3:128
vlarb_low  0:32 1:32 2:48  3:64

Guidelines:

• Keep the number of active VLs modest (4–8). Ensure all HCAs/links support the count you choose.

• Use distinct VLs for classes that must not interfere. Coalesce low-priority classes on the same VL if you’re VL-limited.

5) Apply & persist

Ensure your distro loads the same opensm.conf and qos-policy.conf on boot. On systems with multiple HCAs/SM instances, anchor each OpenSM to a GUID/port and reuse the same policy file.

Verification / Troubleshooting

• Check SM picked up QoS: review opensm.log for policy parsing, SL2VL, and VLArb tables.

• Resolve paths with SA:

  • Get a PathRecord with an explicit Service ID to see the returned SL (and MTU/rate). Using saquery (part of infiniband-diags):

      saquery --sgid-to-dgid <sgid>:<dgid> --service-id 0x0000000000003010 --pkey 0x7fff --sl
    

• Confirm on endpoints: log the CM service_id and the final QP SL your app sees during connection events.

• Port/VL counters: use perfquery/ibqueryerrors to confirm traffic hits the expected VLs; congestion on the wrong VL hints at SL2VL mismatch.

Putting It Together

Trace the path: RDMA-CM port → Service ID → Subnet Manager policy → SL → VL The path from application to QoS is:

RDMA-CM port → Service ID → Subnet Manager policy → SL → VL

The application simply binds to a port; RDMA-CM turns that into a Service ID; the SM maps Service IDs to SLs; the fabric maps SLs to VLs. QoS enforcement is transparent to the app.

Practical Notes

• With IPoIB: CM addresses look like normal IPs, but they resolve to GIDs under the hood.

• Without IPoIB: you can still use RDMA-CM with GID-based addresses; port numbers still become Service IDs.

• If no Service Record matches, the default SL (often 0) is used.

Conclusion

RDMA-CM is the bridge between socket-like connection setup and InfiniBand’s fabric-level QoS. It hides GIDs and QP details from the application, while exposing a port number abstraction that the Subnet Manager can map to Service Levels. This is what allows administrators to classify traffic by port ranges and enforce fabric QoS without requiring applications to manipulate QP attributes directly.

Edit: post now ends on an optimistic note.

Basic IB Concepts

Simple Case: One Subnet

• Subnet Manager (SM): manages the subnet (thx). Like DHCP + ARP + more things.
• lid: like IP address, assigned by subnet manager, you can ping it, send to it, whatever.

$ sudo ibstat
CA 'qib0'
		...
        Port 1:
                State: Active
                Physical state: LinkUp
                Base lid: 475
                SM lid: 10
				...

SM is lid=10, and this node has a HCA (NIC) with lid=0.

Complex Case: Multiple Subnets

Here routing comes into the picture. Won’t go into details, but simple/sloppy version:

• guid: u64: like MAC address, unique per device.
• gid: u128: globally unique ID, sm_id: u64 + guid: u64

Switches maintain routes indexed by sm_id, hosts send packets to gids, switches look up the relevant destination and forward. Not relevant to us going forward.

Miscellaneous

• Service Level (SL): a 4-bit priority carried by a packet
• Virtual Lane (VL): hardware-level egress lanes over a link, these actually provide QoS Isolation
• SL2VL: a table mapping SLs to VLs. SLs confer QoS properties only if they are mapped to separate VLs, and as per specific configuration in VLArb (the VL arbitration tables)
• Subnet Management Packet (SMP): used by SM to configure and query fabric components (ports and switches)
• Management Datagram (MAD): management message format, UMAD is the linux interface to send MAD packets
• Subnet Administrator (SA): a management service that answers queries (part of the Subnet Manager?)
• OpenSM: an open-source software-based subnet manager (like dnsmasq for DHCP). Fabrics may have proprietary ones.
• VL15: a dedicated VL for SMP traffic (QP0: a special queue pair dedicated for it?)

How Many VLs Do I Have?

$ smpquery portinfo <lid> <portnum>
ibwarn: [1006476] mad_rpc_open_port: can't open UMAD port ((null):0)
smpquery: iberror: failed: Failed to open '(null)' port '0'

$ sudo smpquery portinfo <lid> <portnum>
CapMask:.........................0x7610868
                                IsTrapSupported
                                IsAutomaticMigrationSupported
                                IsSLMappingSupported
                                IsSystemImageGUIDsupported
                                IsCommunicatonManagementSupported
                                IsDRNoticeSupported
                                IsCapabilityMaskNoticeSupported
                                IsLinkRoundTripLatencySupported
                                IsClientRegistrationSupported
                                IsOtherLocalChangesNoticeSupported
VLCap:...........................VL0-1
VLHighLimit:.....................0
VLArbHighCap:....................16
VLArbLowCap:.....................16
VLStallCount:....................0
OperVLs:.........................VL0

Takeaways:

1. SLs/VLs are supported (CapMask has IsSLMappingSupported).
2. VLCap suggests that two virtual lanes are possible (VL0 and VL1)
3. OperVLs suggests that only one VL is configured, operational, active, VL1 is not configured.

What Is The Configured QoS State?

$ sudo smpquery sl2vl 475 1
# SL2VL table: Lid 475
#                 SL: | 0| 1| 2| 3| 4| 5| 6| 7| 8| 9|10|11|12|13|14|15|
ports: in  0, out  0: | 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|

$ sudo smpquery vlarb 475 1
# VLArbitration tables: Lid 475 port 1 LowCap 16 HighCap 16
# Low priority VL Arbitration Table:
VL    : |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |
WEIGHT: |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |
# High priority VL Arbitration Table:
VL    : |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |
WEIGHT: |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |

Takeaways:

1. sl2vl says that we have 16 SLs, but all are mapped to VL0, so no QoS.
2. vlarb is all VL0. There are two levels of priority classes. First level is vlarb_high and vlarb_low. Second level is intra-vlarb, between (sl, weight) entries. The relative weight of vlarb-high/low is in VLHighLimit.

Configuring QoS in OpenSM

You edit /etc/opensm/opensm.conf with something like this:

# enable QoS
qos TRUE

# all prefixes are qos_
# specific sub-prefixes: qos_ca_, qos_rtr_, qos_sw0_, qos_swe_
# (for specific config for CAs, routers, switch port 0's, and switches)

qos_max_vls 2
# send this many from high-priority first (255 ~ infinite priority)
qos_high_limit 255
# intra-high priority
qos_vlarb_high 1:192, 2:128, 3:64
qos_vlarb_low 0:64
# SLs [0, 4) are VL0, [4, 8) are VL1
qos_sl2vl 0,0,0,0,1,1,1,1

The policy for relative weights between the high-priority table and the low-priority table is not clear to me. That is, what exactly does qos_vlarb_high do.

Note: https://www.mail-archive.com/lustre-discuss@lists.lustre.org/msg04092.html is a good description of the qos_high_limit. It seems to me that the high class preempts the low class up to the high limit (preempts not mid-packet, but otherwise).

There’s also ULP-based QoS I think (ULP: Upper Layer Protocol). Can maybe target service IDs for MPI, Lustre etc.

Service IDs

• Lustre etc. can register a Service Record with SA, containing (Service ID, server LID, service name)
• RDMA-CM (Connection Manager) is involved somehow
• RDMA-CM divides the SID space into port spaces. RDMA_PS_TCP is a 16-bit namespace.

So SID can be RDMA_PS_TCP << 16 | service_port, or RDMA_PS_IB << 16 | qpn.

$ sudo saquery -S
<nothing>
$ sudo saquery
<bunch of node records>
# On Lustre root
$ cat /sys/module/ko2iblnd/parameters/service
987

So it seems that Lustre’s SID is RDMA_PS_TCP << 16 | 987

$ rdma resource show cm_id | grep LISTEN
link qib0/1 cm-idn 0 state LISTEN ps TCP comm [ko2iblnd] src-addr 10.94.xxx.yyy:987 dst-addr 0.0.0.0:0
# P=987, 0x106 is the `RDMA_PS_TCP`
$ P=987; printf '0x%016x\n' $(( (0x106<<16) | P ))
0x00000000010603db # Service ID for Lustre

This also shows 987 for Lustre. This maps Lustre to SL1.

# /var/cache/opensm/qos-policy.conf

qos-ulps
  default : 0
  any, service-id 0x00000000010603DB : 1
end-qos-ulps

QoS in PSM

• Use env variable IPATH_SL.

if (!psmi_getenv("IPATH_SL", "IB outging ServiceLevel number (default 0)",
		PSMI_ENVVAR_LEVEL_USER, PSMI_ENVVAR_TYPE_LONG,
		(union psmi_envvar_val) PSMI_SL_DEFAULT,
		&env_sl)) {
opts.outsl = env_sl.e_long;
}

// This seems to be older code for PSM v1.07 to 1.10. Head is 1.16
// No need to set VL in head, will use 
#if (PSM_VERNO >= 0x0107) && (PSM_VERNO <= 0x010a)
{
  union psmi_envvar_val env_vl;
  if (!psmi_getenv("IPATH_VL", "IB outging VirtualLane (default 0)",
		   PSMI_ENVVAR_LEVEL_USER, PSMI_ENVVAR_TYPE_LONG,
		   (union psmi_envvar_val)0,
		   &env_vl)) {
opts.outvl = env_vl.e_long;
  }
}
#endif

QoS in libfabric/Mercury

Traffic classes are in fi_tx_attr->tclass (u32): https://ofiwg.github.io/libfabric/main/man/fi_endpoint.3.html

Mercury (since v2.4):

hg_opts.na_init_info.traffic_class = (uint8_t)(desired_sl << 5); // SL -> tclass

Maybe something here: https://docs.nvidia.com/doca/sdk/rdma%2Bover%2Bconverged%2Bethernet/index.html

Things I’ve established so far:

1. I’m interested in verbs; ofi_rxm
2. ofi_rxm just passes tclass to the core provider (which is verbs)
3. I can’t find out what verbs does with it. It seems to use both ibv APIs and rdma-cm APIs. Looks like I may be looking for rdma_set_option(RDMA_OPTION_ID_TOS). But nothing calls that.

Update: 20250923

QoS in libfabric verbs provider: The verbs provider implements QoS through InfiniBand’s native Service Level (SL) mechanism. However, unlike other libfabric providers (e.g., CXI), verbs has no traffic class (tclass) support. The SL is essentially fixed at endpoint creation time and cannot be modified at runtime without recreating the address handle, which would require modifying the libfabric source code.

• SL set in AH: prov/verbs/src/verbs_dgram_av.c:65 - where ah_attr.sl gets the SL value during address handle creation
• SL source: prov/verbs/src/verbs_ep.c:840 - where endpoint gets SL from subnet manager default port_attr.sm_sl
• No tclass support: prov/verbs/src/verbs_ep.c:358 - vrb_ep_setopt() function only handles CUDA options, no tclass
• Compare CXI: prov/cxi/src/cxip_ep.c:1148 - shows how other providers handle FI_OPT_CXI_SET_TCLASS
• Traffic class constants: include/rdma/fabric.h:357 - the FI_TC_* enums that verbs doesn’t use

Update: 20250923 +10 mins

So libfabric has betrayed us, but can we assign the flow a service level anyway, using other classification mechanisms, like that ULP stuff?

Our flow is that we use Mercury to bind to a verbs; ofi+rxm endpoint, and exchange those addresses with friends. What does that address look like?

Our mercury address: ofi+verbs;ofi_rxm://10.94.3.29:56504

Spicy.

$ rdma resource show cm_id | grep LISTEN
...
link qib0/1 cm-idn 17325 state LISTEN ps TCP pid 1344261 comm controller_main src-addr 10.94.3.29:56504 dst-addr 0.0.0.0:0

So it seems that this does go through RDMA-CM. So if we ask Mercury to bind on specific port ranges, I think we can still assign them a QoS using a port range in the ULP thing.

Also Read

• https://hpcadvisorycouncil.atlassian.net/wiki/spaces/HPCWORKS/pages/1178075141/Understanding+Basic+InfiniBand+QoS
• https://hpcadvisorycouncil.atlassian.net/wiki/spaces/HPCWORKS/pages/1177878529/Getting+Started+with+InfiniBand+QoS
• https://web-docs.gsi.de/~vpenso/notes/posts/hpc/network/infiniband/subnet-manager.html
• https://docs.nvidia.com/networking/display/mlnxofedv51258060/opensm
• https://docs.nvidia.com/networking/display/mlnxofedv461000/qos+-+quality+of+service

Lustre discussions:

• https://wiki.whamcloud.com/display/LNet/Lustre+QoS – see this!
• https://groups.google.com/g/lustre-discuss-list/c/n6sdj-e5LNA
• https://www.mail-archive.com/lustre-discuss@lists.lustre.org/msg04092.html
• https://admire-eurohpc.eu/wp-content/uploads/2023/12/Lustre-QoS-Barcelona-GA-Dec-2023.pptx.pdf

Intel Fabric Suite (configrms Lustre ServiceID)

• https://www.intel.com/content/dam/support/us/en/documents/network-and-i-o/fabric-products/Intel_OP_FabricSuite_Fabric_Manager_UG_H76468_v15_0.pdf

Any autonomous system must, by definition, take decisions. It is rational to consider the utility of the decision space and pick a nice point. We draw on our experience and simulate the impact of different decisions in our head, and then we pick one. Is that it?

No, right? Decisions are not irrevocable. You can adapt. If you accidentally oversped, you can apply brakes.

Predictive decisions rely on implicit or explicit modeling. Models are, in the best case, an nth-order approximation of a complex system, with ideally a decent predictive value and a low residual. Intuition, experience, and wisdom are also just models.

Models are by definition imperfect—a perfect model would also necessitate a perfect simulation of all reality. But they can be, and are, more or less imperfect. We all start off with more imperfect models, and (unless maladaptive) use the residuals from using their predictions to refine them. The decision loop, therefore, may be defined to have these components:

1. Observe inputs, and compute decision = f(input, intent)
2. Observe the impact of the decision and compute a course correction if necessary
3. Use the residual to update your hypothesis

Or a more formal attempt:

model: m_init   # Our initial model
intent: i_init  # Some mysterious missing ingredient

while Alive:
  reality = input()
  decision = f(reality, intent, model)
  impact = reality.apply(decision)
  error = intent - impact
  model.update(error)
  intent.update(reality, impact)

What Is An Optimal Decision?

It is tempting to then say that the quality of the initial decision does not matter, because it can always be updated. That models the system as a stochastic process, and is true insofar as that holds/is a reasonable approximation. But that is not the case: human lifetimes are finite and there is a time cost and material cost of bad decisions, there is no recovering from jumping off a cliff, prior actions also affect other agents’ models (what we call reputation or trust).

It may then be tempting to say that you want the best model before you take any decisions. But models are by definition updated by taking decisions. Scheduling of decisions is a useful escape hatch/degree of freedom: you may decide to read a book to have a better model before taking a decision with a bigger modeled impact. Scheduling is perhaps a meta-decision: choosing to explore (deferring the big decision) rather than exploit.

Deferring is useful insofar as the marginal improvement in your model outweighs the opportunity cost of delaying a decision. Two examples of instances where deferring is bad:

1. There exists no decision that meaningfully enhances the utility of the deferred decision
2. The opportunity cost of deferment is large

Connections With Standard Terminology I think open-loop and closed-loop control in control theory are not appropriate parallels. What I am describing is maybe an adaptive closed-loop system? But an autonomous, adaptive, completely programmable closed-loop system where decisions update both the model and the intent. Or just intelligent and conscious life?

Why Is This Useful

How to design systems. If a system requires some information that is, by definition, not available at bootstrap, adaptive mechanisms are necessary to solve the problem.

Autonomous intelligence, maybe? With current ML models, backpropagation is a progressive hypothesis refinement loop, but without an intent. And then model weights are frozen and they are tasked with prediction. But being static, they are not refined, unless we account for periodic batch refinement that is conducted with additional data gathered since the previous run.

Are models sufficiently adaptive? Is it possible that even with static weights, they learn to simulate memories etc. within those weights? What are the limits of computations that an algorithm running on top of a static machine can express, versus those of an algorithm that can mutate the machine itself?

Services in 3FS

• Cluster manager: highly available, manages membership, config etc (uses zookeeper?)
• Metadata service: uses FoundationDB
• Storage service (chunk store?)
• Client (two implementations: FUSE-based, and a more performant one.)

Metadata in 3FS

The Ongoing Metadata Debate

Whether we need object stores or filesystem semantics has been a topic of conversation recently². My own views so far:

• I buy the argument that data accesses during a training run do not need a namespace. Maybe the namespace can be batch-updated later on, if it gets in the way and does not provide the required performance.
• I feel that the usability argument still applies. At some point, you need to think about what does all the data mean, what to keep around, what to prune. There are a million reasons – legal, regulatory, financial, just basic IT maintenance that require metadata. Long-term, it is easy to think about things as a group or a hierarchy.

I think performant fileystem namespaces is doable. It may get needlessly hard if you tie yourself to a spec or a standard. That is because this is in flux, and we do not yet have unanimous agreement on what the standard should look like.

The 3FS Approach

They use a key-value model for their metadata, built on top of FoundationDB. Each key represents an inode.

FoundationDB, as an aside, is a very interesting database. It is like a distributed RocksDB with support for strict serializability in transactions. This is interesting to me — normally you think of KV stores as single-node storage backends for DBMSes to use. Transactions are implemented at a SQL level. FDB implements transactions at the KV level, but is supposed to be a lower-level database you layer on top of. (Aside: I wonder what the end-to-end principle says about the correct layering). It is like a persistent distributed software transactional memory, that tries to be lock-free, with optimistic and multi-version concurrency control.

• File inodes are INOD<inode_id>, mapped to ownership, permissions, times., and chunk information etc. for files. Interestingly, they encode inode_id in little-endian to spread inodes across FDB nodes. Directory inodes begin with DENT, have bidirectional links to parents to ensure no loops.needs to be and should be super-fast.
• fstat, lookup, listdir etc. invoke read-only txns. crate, link etc. invoke read-write txns.

State in 3FS

3FS metadata stores are stateless. Any operation is a self-contained transaction.

The metadata service does store file descriptors for files opened in write mode, but not in read mode. This helps with bootstrapping training jobs. This IMO is super important!! Any parallel job, when bootstrapping, needs to retrieve a large number of files in read-only mode. This should be super-fast, but is not so with Lustre. I vaguely recall someone mentioning that they had to hack on a cache on top of Lustre to push .so libraries to MPI ranks, and thinking that this feels like a fundamental flaw in what is supposed to be a parallel filesystem.

Question: how do you handle a write request for a file that is currently open in read-only mode? Ideally you need to track the readers and revoke their lease, but that requires maintaining state at the metadata layer? I think that’s done by tracking versions?

Chunk Store

This is relatively straightforward. It is per-node, and comes with a RocksDB instance for chunk metadata. Chunk metadata is cached in-memory. They have 11 chunk sizes from 64 KiB to 64 MiB. I don’t think this is a big deal but metadata shouldn’t be too much, even with 64KiB chunks, it is a little surprising that they went for this number of chunk sizes. But eh. Updates are CoW – old chunks remain valid until update completes.

Chain Replication

They use CRAQ (Chained Replication with Apportioned Queries) for replication. This is just a fancy way of saying that any replica can respond to a read request, with the catch being that you may get a stale version. A node may have a committed version of a block and a pending version. If so, it lets the requester know the existence of both versions, and the decision on whether to tolerate staleness or try again is left to the requester. Note that this is different from the case where a chain replica does not even know of the existence of an update – I don’t know if stale reads are tolerated in that case (or maybe version numbers are used to read a consistent snapshot).

Zero-copy in FUSE!?

They use FUSE for clients, so as to not deal with debugging kernel panics. They define an interface called USRBIO, for zero-copy interaction between the userspace and the FUSE layer. USRBIO is inspired by io-uring (or the Verbs API for that matter). The FUSE process manages Verbs-registered memory and submission/completion rings, and does the dispatch etc. These zero-copy ops are only used for the data path, and metadata ops still go through regular FUSE APIs.

Misc

• Codebase is mostly C++ with some Rust.
• They use P³ to verify their protocols. P is apparently a formal verification language for event-driven distributed services. Very cool!
• They seem to use flatbuffers for serdes and their own RDMA/RPC wrappers on top of the verbs interface. There’s a lot of code there and I just briefly skimmed through it.
• They heavily use folly, Facebook’s assorted library of C++ abstractions (including coroutines).

Epilogue

Not sure what to make of this. Writing a parallel filesystem is a massive undertaking. While metadata bottlenecks is a well-known problem in parallel filesystems, offloading it entirely to a DB like FoundationDB is an interesting choice. I have been trying to revisit how Colossus/Tectonic manage their metadata, and it is Bigtable and ZippyDB respectively. Ceph is pretty much the only open-source filesystem to support multiple metadata servers (Lustre is unwieldy in multiple ways). 3FS is optimized for RDMA fabrics over the data path, and makes subtle choices (like cheap metadata reads) to optimize for a large parallel job bootstrapping. I remember reading somewhere that it is designed for small random accesses, and I don’t really see why that is the case. I will end with the caveat that there are a lot of things to keep straight in distributed/parallel filesystem discussions, I don’t really work on these, and this post probably has a bunch of mistakes.

(Oh their design notes do say that they enable random access to training samples, I don’t see how they optimize for random accesses, maybe they mean that their system will transfer partial chunks and not do aggressive prefetching etc?)

References

Goal for this post: let us try justifying SQL. Why it exists, why is it in a specific form etc.

SQL vs Relational Algebra

I think SQL is made-up and non-essential. It is syntactic sugar over relational algebra. Other declarative interfaces may be more convenient abstractions — I personally prefer dataframes and chaining of expressions.

If you think of the query plan as a dataflow tree leading up to a root, SQL is essentially a traversal over that tree, like in-order, pre-order etc. The order SQL follows, specifically, may be called a Weird Arbitrary Traversal (or WAT). That the query planning infra evolved to reflect dataflow is not surprising, data movement is expensive, be it across functions or machines, and the more data you discard closer to the source the better off you are.

These pedantic distinctions are important, because they enable us to reason about

On the Universality of Relational Algebra?

Okay so why does relational algebra exist in the form it does?

As a lower bound on this argument, a query interface could just be say… C. You write a blob, machine runs blob, you get output.

We do declarative interfaces to make life easier given certain domain-specific constraints. In the relational world, it is:

1. Leveraging the relational/tabular data model.
2. Leveraging distributed cluster resources.
3. Leveraging fine-grained scheduling, concurrency etc.
4. Getting easy access to indexes etc.
5. Enabling reduced data movement by exposing primitives that capture data flow
6. Enabling query optimization by exposing primitives that make it easier to reason about ordering and statistics
7. Enable reasoning about partitioning and shuffling using key constraints and joins

Do Relational Operators Do These Things?

To a large extent, yes. That is why relational algebra endures. SQL just pretends to be inseparable from that and most people do not ask too many questions.

You always need escape hatches. That is why UDFs exist. But beyond a point, they hinder the query planning infra’s ability to reason about what the blob does.

Who Deserves To Be In The Club?

What operators should be in this club?

• Can we justify the existence of all existing operators?
• Are there new ones that should be there but are not?
• Under what models do we need radically different solutions?

Relational algebra checks out as it is essentially set theory. Given a table, you can slice it horizontally or vertically. Indexing helps with horizontal slicing. Vertical slicing is harder in row-based stores but column-based stores have solutions. Selection and projection map nicely to dataflow, indexing etc. Division seems to be the only relational operator not directly used in query plans.

Where Am I Getting At?

There exist data models (such as multidimensional array-based scientific data) that are not relational. Graphs are one example. Nothing here is really new.

The question I am trying to answer is: what is the set of ideas that lead you to the perfect operators for these domains? Do they exist? Are they in conflict with relational algebra? Or can they be members of the extended family?

Note to self: read referenced material ¹²³⁴

References

This post will explore the following thesis:

1. All networks and variants are tradeoffs.
2. If there are tradeoffs, there is no permanent winning. You are always tuning to chase the window.

Disclaimer: this could be the most obvious statement in the world. Or maybe deep within the symbolic mines of queuing and scheduling, someone has said something that disproves all of it. Or maybe that that statement is obvious but the arguments below are flawed. I have no idea — this blog does not come with any guarantees for rigor or scrutiny.

Context: Thinking About CBFC vs PFC

The thing in the back of my mind recently has been — why does everyone complain about PFC and RoCEv2, and why do they not complain about Infiniband and CBFC (Credit-based Flow Control). Why could RoCEv2 not adopt CBFC? What do you gain and what do you lose? Do you still need higher-level congestion control with CBFC?

CBFC Is Not A Magic Pill

An upper bound to CBFC goodness is much easier to establish than its precise extent.

CBFC is not a magic pill. It can still lead to credit loops, head-of-line blocking, buffer overflows, packet loss emanating from it, and so on. It is also amenable to tuning: routing, topologies, credit management schemes etc. to reduce the likelihood of bad things happening.

CBFC absolutely provides no theoretical guarantee (I’m not talking about a bounds-type argument: under X flows and Y buffer sizes you prove a Z\% upper bound on packet loss). Also not everyone means the same thing when they talk about CBFC — the Infiniband spec on this is not very prescriptive — my understanding is that it describes packet formats, and some basic mechanics, but still leaves a lot of room for vendors to do better.

Guarantees Are Expensive

You could provide a lot of guarantees by implementing a network as an all-to-all network of separate links. That is essentially what a crossbar is. But we want to optimize for cost. So we do tiers and routing and non-uniform bandwidth and all that stuff. You lose out on worst-case performance but that’s the only way to get systems to scale.

Turns out that this applies for everything — including tail latencies, jitter etc. The only way to guarantee that you will move a packet in a predictable timely manner is to explicitly carve out room for that packet on the path before sending it out. If you care about maintaining a high utilization or cost or scalability, this is bad.

Given a guarantee, you can always make an argument that “we can relax this guarantee a little and get a lot more performance in the common case.” This is statistical multiplexing or oversubscription or optimistic concurrency control or whatever. These are incredibly important — these thoughtful relaxations are the reason that systems work and can be built for the prices that’re built for. But it is always nice to be explicit about what you are losing in return.

Overloading Creates More Risks

As the goal of this exercise was to understand CBFC, let us compare it with sender-driven mechanisms. Sender-driven congestion control overloads certain signals to infer network state — RTT and packet loss seem to be two of them. This works until something happens that changes the meaning of these numbers — bufferbloat being one example. An advantage, in principle, of CBFC is that it is explicitly communicating actual network state, so theoretically, it should be robust to some such issues. I do not know if that is actually realized in practice.

This is in no way trashing sender-driven mechanisms. A good system is one that gets the job done — clean conscience and mathematical elegance are the domains of whiteboard hoggers. The end-to-end principle and this layered approach to TCP/IP has taken us a long long long way. Overloading those signals is the practical thing to do. (Also note that I’m conflating flow control with congestion control — CBFC is very much the former, but I guess it dictates and comes bundled with a different congestion control in IB with ECNs and injection throttling).

Expectations for Future Congestion Control

This post is a precursor to me trying to understand all the cool new things in datacenter fabrics — Omnipath, Slingshot, UltraEthernet, maybe Globally Scheduled Ethernet?

If these arguments hold, none of these systems’ mechanisms will be perfect. There will always be cases where Bad Things$^{TM}$ happen. Part of the argument is that that may be okay, and an evaluation of their properties must be holistic.

Some Degree Of Tuning Is Okay?

• I can not pick an outfit that works in all weathers. I am destined to keep tuning.
• Some tuning is worse than others. If I wore an outfit that had only one shoe, I would have to hop and keep swapping my shoe from one foot to the other.
• An outfit that covers the expected swings over one day is sufficient for an outfit.
• A bag that covers the expected swings over one week is sufficient for a short trip.
• A wardrobe that covers the expected swings over the year in Pittsburgh is sufficient for most other cases.

What are the takeaways for systems?

• We are destined to tune. We should plan for that.
• It would be nice if we understood systems in terms of their coverage of design space.
• Nothing in this discussion rules out the existence of systems that cover strictly more of the design space than others. Probably someone has and/or will prove that certain aspects are pareto-optimal. But it’s still a small sheet over massive legs.

Chasing The Global Optima

Network sharing and scheduling mechanisms are probably by far the most well-studied and deployed examples of a distributed system trying to construct/reach a global optimum, and both mathematical and practical arguments of the challenges that comes with.

All systems would benefit from better decisions. Better decisions are enabled by a better view of the system state. But constructing this system state may be prohibitively expensive. It may also be a moving target. It will definitely have some uncertainty bounds because physics. It may also be possible that explicitly constructing the system state is not necessary — we can apparently go quite a bit with individual actors acting on simple rules. (Something about game theory and cooperative games comes to mind but I’m already wayyyy past my depth here).