Edit: I prompt-engineered some LLMs into generating this. Same reliability caveats as the rest of the post apply.

Losslessless: The Bedrock of RDMA

RDMA originated on a natively lossless fabric (Infiniband). Since then, we have tried to graft it into inherently lossy fabrics, with brittle outcomes. We like losslessness for the same reason we like strong consistency: it is nice and simple if things just work and are predictable.

1. Every network becomes lossless at low utilization.
2. CBFC is the best way we know to build a lossless network at higher traffic volumes. PFC is very fragile, intrinsically so.
3. CBFC and PFC represent two fundamentally differing approaches to utilization management: proactive and reactive, with their own tradeoffs. The cost of a proactive approach is “imposing flow control when it was unnecessary”, leading to potential underutilization.
4. There is a fundamental tradeoff between network utilization and loss guarantees. For any given lossless network, you can afford to be a little speculative and increase median usage. The null hypothesis is that this increases loss probability. Unless reasoned all the way through and demonstrated to not exist or not be a factor, it must exist.
5. We like lossless networks because in certain conditions, the cost of loss to higher layers is prohibitive. We should not design lossless schemes that dispatch speculatively without reasoning about why we wanted a lossless scheme in the first place, and why that speculative dispatch does not create problems we wanted to avoid¹.

Everyone Likes Most of Infiniband

Let us disaggregate infiniband into a series of features.

Likes:

1. Everyone likes CBFC[^1,2,3]
2. Everyone likes the performance benefits of RDMA, if not the complexity of memory registration and coding verbs.
3. We like the cache-friendy access patterns enabled by verbs as an interface. The TCP/IP stack, as implemented, has been noted to force cache-unfriendly behavior, creating utilization bubbles in the precious PCIe link². verbs, used along with capabilities such as SRQs, multi-receive, and hugepages enable intrinsically contiguous accesses.

Dislikes:

1. Everyone dislikes its difficult relationship with the broader datacenter network. It is not a part of the broader network, but a separate sub-network like NVLink or CXL.
2. Many folks dislike its scalability constraints (connections need QPs need NIC SRAM).

What We Really Need: RD + Ethernet Compatibility

Case Study: QLogic

QLogic already had this insight 15 years ago. QLogic is not super well known, and I think it is unfortunate. Many aspects of their approach are highly relevant to some debates we have had since.

QLogic Truescale was the precursor to Intel Omnipath, which was the precursor to Cornelis Networks. Instead of offload to a smart NIC, they wanted onload: CPU-mediated communication for scalability. It was a verbs-compliant fabric that natively spoke a different interface called psm.

Truescale was a messaging-oriented fabric. I think of it as RDMA UD with large datagram sizes. It injected small payloads inline into a packet, and large payloads were DMA’ed on both the send and receive paths using transient connection mechanisms called TID/TIDflows.

From what I understand, PSM had CBFC but no link-level retry (LLR). It had a software-based go-back-N-style recovery. I think link-level retry requires a smarter HCA, which they were trying to avoid, at least for the first generation. I have observed software-based retries in that fabric to add an extraordinary amount of jitter, when triggered, but it is triggered by some residual bugs and high timeouts, and not by packet drops.

Case Study: SRD in Amazon EFA

The variable-sized datagram design point remains compelling. Amazon built a fabric called Elastic Fabric Adapter (EFA) that presumably adds LLR to UD. I don’t know much about it so I’ll keep it short.

RDMA vs RPCs: A False Dichotomy?

I’m probably wading into a decade-old debate at this point, but it’s not my fault that I was just a swaddling infant a decade ago.

Let us take eRPC³, an iconic paper at this point. The argument I will explore is whether eRPC implements Infiniband in software over lossy ethernet, and whether while a solid step forward for lossy ethernet, its design just strengthens the argument for real RDMA/infiniband.

1. eRPC likes losslessness. They create those conditions by having session credits and limiting dispatched traffic to network BDP. Session credits resemble an end-to-end version of CBFC, something also tried in ⁴.
2. eRPC likes contiguous message buffers, an event loop for short messages, and memcpy’ed contiguous buffers for large messages, with memcpy orchestrated by CPU. This feels very close to the workflow I employ in my current codebase using the RDMA-capable Mercury RPC library, except using CPU cycles instead of RDMA.

I wonder what is the difference between “we found a lossless regime in the network and designed a system optimized for a lossless common-case” and “we engineered a lossless regime in the network, and this proves the value of a genuinely lossless fabric.”

I also wonder about the value of pushing retries to the application layer when the ultimate behavior you want is lossless. There are plenty of applications that tolerate lossiness end-to-end, and it makes sense for the lowest common denominator to not include retries for general-purpose networks. But datacenters are a controlled high-density environment, and maybe it is easier to permit lossy flows as a second-class citizen in a lossless fabric than the other way round?

Also, I think scalability criticisms of infiniband are hardly the killer argument they are made out to be. Truescale is an example of a production fabric that already solved those challenges, and software-level solutions such as RxD exist in libfabric (multiplexing RDMA over UD). RDMA is not infiniband, and there are multiple CBFC-based approaches to lossless fabrics.

Next, I think RDMA doesn’t preclude the ability to do two-sided communication. I think it is not completely accurate to think of RDMA as disaggregated memory. That is, RDMA is not CXL. An RDMA-capable node is essentially still a request/response-based server. Maybe it is useful to mentally decouple operations into metadata and data, and use RDMA for the data path (common criticism is unnecessary RTTs for pointer chasing). Mercury⁵, with providers for libfabric etc, provides both messaging-style semantics for small messages and verbs-enabled RDMA via a bulk interface for large messages.

Finally, I think the end-to-end argument does not preclude merging of layers for performance. I think it is a neat guiding principle for what to do in the absence of competing concerns, but it does not preemptively override the presence of competing concerns.

What’s Next?

I want to understand HPE Slingshot and UltraEthernet better, and maybe also start dabbling with CXL.

Side note: do you know PCIe uses CBFC? I think the gap between an interconnect, a fabric, and a network is smaller than we think!

References

Updated on 20251008 and later on 20251014 to present a version that may age better.

Introduction

RDMA-CM (RDMA Connection Manager) is the control-plane layer for RDMA. It exists to make connection setup look and feel like sockets: applications can bind to addresses, listen on ports, and accept or connect without worrying about raw InfiniBand identifiers. Underneath, RDMA-CM exchanges the connection metadata, programs the QPs, and hands back an endpoint ready for verbs. This is distinct from the verbs API, which only covers the data plane.

Background: QoS in InfiniBand

InfiniBand provides Quality of Service using Service Levels (SLs). An SL is a small integer carried in each packet’s header. The subnet manager maps SLs onto Virtual Lanes (VLs), which are independent link-level flows with separate buffering and flow control. By assigning flows to different SLs, the fabric can prioritize some traffic or keep classes of traffic from interfering with each other. SL is selected when a QP is created, and remains fixed for that QP’s traffic.

RDMA-CM Basics

RDMA-CM exposes a socket-like API:

• The central handle is an rdma_cm_id, analogous to a socket fd.

• Each cm_id belongs to a port space, such as RDMA_PS_TCP (connection-oriented) or RDMA_PS_UDP (datagrams).

• Applications use rdma_bind_addr, rdma_listen, rdma_connect, etc., in the same way they would with sockets.

What looks like an IP+port pair in RDMA-CM is just a wrapper. With IPoIB enabled, the “IP” is the interface address (e.g. 10.94.3.105), which RDMA-CM resolves to a GID. Without IPoIB, you still can create connections, but the address is directly a GID (usually expressed in IPv6 link-local form). In both cases, the binding resolves to a specific HCA port and GID.

The handshake itself is a control-plane exchange: on InfiniBand it uses CM MADs over QP1, on RoCE/iWARP it uses UDP/TCP messages. This is how QP numbers, PSNs, MTU, and SL hints are exchanged.

RDMA-CM and Service IDs

The key to QoS integration is that RDMA-CM maps the familiar notion of port numbers into InfiniBand Service IDs. A Service ID is a 64‑bit identifier that the Subnet Manager understands. In RDMA_PS_TCP, the port number you bind is encoded into the Service ID automatically. This provides a clean classification hook: flows created on different port ranges correspond to different Service IDs.

Subnet Manager QoS Policies & Setup (OpenSM)

OpenSM can enforce QoS by mapping Service IDs (derived from RDMA-CM port numbers) to Service Levels (SLs) and then to Virtual Lanes (VLs).

1) Enable QoS in OpenSM

Edit /etc/opensm/opensm.conf (or the file you pass with -f) and set:

qos TRUE
qos_max_vls 8              # adjust to your ASIC/link support (e.g., 4 or 8)
qos_policy_file /etc/opensm/qos-policy.conf

You may additionally pin SL→VL and VL arbitration globally here if you want static tables in the options file (advanced fabrics often keep these in the policy file instead).

Restart or start OpenSM with QoS active:

systemctl restart opensm    # distro services
# or manually
opensm -Q -f /etc/opensm/opensm.conf
# (-Q/--qos is equivalent to qos TRUE; -Y sets an explicit policy file path)

2) Define QoS policy (port→ServiceID→SL)

Create /etc/opensm/qos-policy.conf with policy rules. A pragmatic pattern is to carve the RDMA-CM port space into classes and assign SLs per Service ID range. Example with three classes:

# --- Classes ---------------------------------------------------
# Control plane / RPC (low bandwidth, low latency)
service_id 0x0000000000001000-0x0000000000001fff sl 4

# Bulk telemetry / background
service_id 0x0000000000002000-0x0000000000002fff sl 1

# Data path / latency-critical (highest priority)
service_id 0x0000000000003000-0x0000000000003fff sl 6

Notes:

• RDMA-CM encodes RDMA_PS_TCP + port into a 64-bit Service ID. Use contiguous Service ID ranges to represent your port ranges. Confirm the exact mapping in your stack by printing the CM service_id on connection or using SA queries (see verification below).

• If you use IPoIB, classification still hinges on Service ID, not L3 QoS.

3) Map SLs to VLs (isolation / priority)

Still in qos-policy.conf, define SL→VL and VL arbitration so the fabric actually separates traffic:

# Map each SL to a VL. Keep hot classes on distinct VLs.
sl2vl 0 0
sl2vl 1 1
sl2vl 4 2
sl2vl 6 3

# Basic VL arbitration (weights). Keep highest-priority VLs with more credits.
vlarb_high 0:64 1:64 2:96 3:128
vlarb_low  0:32 1:32 2:48  3:64

Guidelines:

• Keep the number of active VLs modest (4–8). Ensure all HCAs/links support the count you choose.

• Use distinct VLs for classes that must not interfere. Coalesce low-priority classes on the same VL if you’re VL-limited.

5) Apply & persist

Ensure your distro loads the same opensm.conf and qos-policy.conf on boot. On systems with multiple HCAs/SM instances, anchor each OpenSM to a GUID/port and reuse the same policy file.

Verification / Troubleshooting

• Check SM picked up QoS: review opensm.log for policy parsing, SL2VL, and VLArb tables.

• Resolve paths with SA:

  • Get a PathRecord with an explicit Service ID to see the returned SL (and MTU/rate). Using saquery (part of infiniband-diags):

      saquery --sgid-to-dgid <sgid>:<dgid> --service-id 0x0000000000003010 --pkey 0x7fff --sl
    

• Confirm on endpoints: log the CM service_id and the final QP SL your app sees during connection events.

• Port/VL counters: use perfquery/ibqueryerrors to confirm traffic hits the expected VLs; congestion on the wrong VL hints at SL2VL mismatch.

Putting It Together

Trace the path: RDMA-CM port → Service ID → Subnet Manager policy → SL → VL The path from application to QoS is:

RDMA-CM port → Service ID → Subnet Manager policy → SL → VL

The application simply binds to a port; RDMA-CM turns that into a Service ID; the SM maps Service IDs to SLs; the fabric maps SLs to VLs. QoS enforcement is transparent to the app.

Practical Notes

• With IPoIB: CM addresses look like normal IPs, but they resolve to GIDs under the hood.

• Without IPoIB: you can still use RDMA-CM with GID-based addresses; port numbers still become Service IDs.

• If no Service Record matches, the default SL (often 0) is used.

Conclusion

RDMA-CM is the bridge between socket-like connection setup and InfiniBand’s fabric-level QoS. It hides GIDs and QP details from the application, while exposing a port number abstraction that the Subnet Manager can map to Service Levels. This is what allows administrators to classify traffic by port ranges and enforce fabric QoS without requiring applications to manipulate QP attributes directly.

Edit: post now ends on an optimistic note.

Basic IB Concepts

Simple Case: One Subnet

• Subnet Manager (SM): manages the subnet (thx). Like DHCP + ARP + more things.
• lid: like IP address, assigned by subnet manager, you can ping it, send to it, whatever.

$ sudo ibstat
CA 'qib0'
		...
        Port 1:
                State: Active
                Physical state: LinkUp
                Base lid: 475
                SM lid: 10
				...

SM is lid=10, and this node has a HCA (NIC) with lid=0.

Complex Case: Multiple Subnets

Here routing comes into the picture. Won’t go into details, but simple/sloppy version:

• guid: u64: like MAC address, unique per device.
• gid: u128: globally unique ID, sm_id: u64 + guid: u64

Switches maintain routes indexed by sm_id, hosts send packets to gids, switches look up the relevant destination and forward. Not relevant to us going forward.

Miscellaneous

• Service Level (SL): a 4-bit priority carried by a packet
• Virtual Lane (VL): hardware-level egress lanes over a link, these actually provide QoS Isolation
• SL2VL: a table mapping SLs to VLs. SLs confer QoS properties only if they are mapped to separate VLs, and as per specific configuration in VLArb (the VL arbitration tables)
• Subnet Management Packet (SMP): used by SM to configure and query fabric components (ports and switches)
• Management Datagram (MAD): management message format, UMAD is the linux interface to send MAD packets
• Subnet Administrator (SA): a management service that answers queries (part of the Subnet Manager?)
• OpenSM: an open-source software-based subnet manager (like dnsmasq for DHCP). Fabrics may have proprietary ones.
• VL15: a dedicated VL for SMP traffic (QP0: a special queue pair dedicated for it?)

How Many VLs Do I Have?

$ smpquery portinfo <lid> <portnum>
ibwarn: [1006476] mad_rpc_open_port: can't open UMAD port ((null):0)
smpquery: iberror: failed: Failed to open '(null)' port '0'

$ sudo smpquery portinfo <lid> <portnum>
CapMask:.........................0x7610868
                                IsTrapSupported
                                IsAutomaticMigrationSupported
                                IsSLMappingSupported
                                IsSystemImageGUIDsupported
                                IsCommunicatonManagementSupported
                                IsDRNoticeSupported
                                IsCapabilityMaskNoticeSupported
                                IsLinkRoundTripLatencySupported
                                IsClientRegistrationSupported
                                IsOtherLocalChangesNoticeSupported
VLCap:...........................VL0-1
VLHighLimit:.....................0
VLArbHighCap:....................16
VLArbLowCap:.....................16
VLStallCount:....................0
OperVLs:.........................VL0

Takeaways:

1. SLs/VLs are supported (CapMask has IsSLMappingSupported).
2. VLCap suggests that two virtual lanes are possible (VL0 and VL1)
3. OperVLs suggests that only one VL is configured, operational, active, VL1 is not configured.

What Is The Configured QoS State?

$ sudo smpquery sl2vl 475 1
# SL2VL table: Lid 475
#                 SL: | 0| 1| 2| 3| 4| 5| 6| 7| 8| 9|10|11|12|13|14|15|
ports: in  0, out  0: | 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|

$ sudo smpquery vlarb 475 1
# VLArbitration tables: Lid 475 port 1 LowCap 16 HighCap 16
# Low priority VL Arbitration Table:
VL    : |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |
WEIGHT: |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |
# High priority VL Arbitration Table:
VL    : |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |
WEIGHT: |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |0x0 |

Takeaways:

1. sl2vl says that we have 16 SLs, but all are mapped to VL0, so no QoS.
2. vlarb is all VL0. There are two levels of priority classes. First level is vlarb_high and vlarb_low. Second level is intra-vlarb, between (sl, weight) entries. The relative weight of vlarb-high/low is in VLHighLimit.

Configuring QoS in OpenSM

You edit /etc/opensm/opensm.conf with something like this:

# enable QoS
qos TRUE

# all prefixes are qos_
# specific sub-prefixes: qos_ca_, qos_rtr_, qos_sw0_, qos_swe_
# (for specific config for CAs, routers, switch port 0's, and switches)

qos_max_vls 2
# send this many from high-priority first (255 ~ infinite priority)
qos_high_limit 255
# intra-high priority
qos_vlarb_high 1:192, 2:128, 3:64
qos_vlarb_low 0:64
# SLs [0, 4) are VL0, [4, 8) are VL1
qos_sl2vl 0,0,0,0,1,1,1,1

The policy for relative weights between the high-priority table and the low-priority table is not clear to me. That is, what exactly does qos_vlarb_high do.

Note: https://www.mail-archive.com/lustre-discuss@lists.lustre.org/msg04092.html is a good description of the qos_high_limit. It seems to me that the high class preempts the low class up to the high limit (preempts not mid-packet, but otherwise).

There’s also ULP-based QoS I think (ULP: Upper Layer Protocol). Can maybe target service IDs for MPI, Lustre etc.

Service IDs

• Lustre etc. can register a Service Record with SA, containing (Service ID, server LID, service name)
• RDMA-CM (Connection Manager) is involved somehow
• RDMA-CM divides the SID space into port spaces. RDMA_PS_TCP is a 16-bit namespace.

So SID can be RDMA_PS_TCP << 16 | service_port, or RDMA_PS_IB << 16 | qpn.

$ sudo saquery -S
<nothing>
$ sudo saquery
<bunch of node records>
# On Lustre root
$ cat /sys/module/ko2iblnd/parameters/service
987

So it seems that Lustre’s SID is RDMA_PS_TCP << 16 | 987

$ rdma resource show cm_id | grep LISTEN
link qib0/1 cm-idn 0 state LISTEN ps TCP comm [ko2iblnd] src-addr 10.94.xxx.yyy:987 dst-addr 0.0.0.0:0
# P=987, 0x106 is the `RDMA_PS_TCP`
$ P=987; printf '0x%016x\n' $(( (0x106<<16) | P ))
0x00000000010603db # Service ID for Lustre

This also shows 987 for Lustre. This maps Lustre to SL1.

# /var/cache/opensm/qos-policy.conf

qos-ulps
  default : 0
  any, service-id 0x00000000010603DB : 1
end-qos-ulps

QoS in PSM

• Use env variable IPATH_SL.

if (!psmi_getenv("IPATH_SL", "IB outging ServiceLevel number (default 0)",
		PSMI_ENVVAR_LEVEL_USER, PSMI_ENVVAR_TYPE_LONG,
		(union psmi_envvar_val) PSMI_SL_DEFAULT,
		&env_sl)) {
opts.outsl = env_sl.e_long;
}

// This seems to be older code for PSM v1.07 to 1.10. Head is 1.16
// No need to set VL in head, will use 
#if (PSM_VERNO >= 0x0107) && (PSM_VERNO <= 0x010a)
{
  union psmi_envvar_val env_vl;
  if (!psmi_getenv("IPATH_VL", "IB outging VirtualLane (default 0)",
		   PSMI_ENVVAR_LEVEL_USER, PSMI_ENVVAR_TYPE_LONG,
		   (union psmi_envvar_val)0,
		   &env_vl)) {
opts.outvl = env_vl.e_long;
  }
}
#endif

QoS in libfabric/Mercury

Traffic classes are in fi_tx_attr->tclass (u32): https://ofiwg.github.io/libfabric/main/man/fi_endpoint.3.html

Mercury (since v2.4):

hg_opts.na_init_info.traffic_class = (uint8_t)(desired_sl << 5); // SL -> tclass

Maybe something here: https://docs.nvidia.com/doca/sdk/rdma%2Bover%2Bconverged%2Bethernet/index.html

Things I’ve established so far:

1. I’m interested in verbs; ofi_rxm
2. ofi_rxm just passes tclass to the core provider (which is verbs)
3. I can’t find out what verbs does with it. It seems to use both ibv APIs and rdma-cm APIs. Looks like I may be looking for rdma_set_option(RDMA_OPTION_ID_TOS). But nothing calls that.

Update: 20250923

QoS in libfabric verbs provider: The verbs provider implements QoS through InfiniBand’s native Service Level (SL) mechanism. However, unlike other libfabric providers (e.g., CXI), verbs has no traffic class (tclass) support. The SL is essentially fixed at endpoint creation time and cannot be modified at runtime without recreating the address handle, which would require modifying the libfabric source code.

• SL set in AH: prov/verbs/src/verbs_dgram_av.c:65 - where ah_attr.sl gets the SL value during address handle creation
• SL source: prov/verbs/src/verbs_ep.c:840 - where endpoint gets SL from subnet manager default port_attr.sm_sl
• No tclass support: prov/verbs/src/verbs_ep.c:358 - vrb_ep_setopt() function only handles CUDA options, no tclass
• Compare CXI: prov/cxi/src/cxip_ep.c:1148 - shows how other providers handle FI_OPT_CXI_SET_TCLASS
• Traffic class constants: include/rdma/fabric.h:357 - the FI_TC_* enums that verbs doesn’t use

Update: 20250923 +10 mins

So libfabric has betrayed us, but can we assign the flow a service level anyway, using other classification mechanisms, like that ULP stuff?

Our flow is that we use Mercury to bind to a verbs; ofi+rxm endpoint, and exchange those addresses with friends. What does that address look like?

Our mercury address: ofi+verbs;ofi_rxm://10.94.3.29:56504

Spicy.

$ rdma resource show cm_id | grep LISTEN
...
link qib0/1 cm-idn 17325 state LISTEN ps TCP pid 1344261 comm controller_main src-addr 10.94.3.29:56504 dst-addr 0.0.0.0:0

So it seems that this does go through RDMA-CM. So if we ask Mercury to bind on specific port ranges, I think we can still assign them a QoS using a port range in the ULP thing.

Also Read

• https://hpcadvisorycouncil.atlassian.net/wiki/spaces/HPCWORKS/pages/1178075141/Understanding+Basic+InfiniBand+QoS
• https://hpcadvisorycouncil.atlassian.net/wiki/spaces/HPCWORKS/pages/1177878529/Getting+Started+with+InfiniBand+QoS
• https://web-docs.gsi.de/~vpenso/notes/posts/hpc/network/infiniband/subnet-manager.html
• https://docs.nvidia.com/networking/display/mlnxofedv51258060/opensm
• https://docs.nvidia.com/networking/display/mlnxofedv461000/qos+-+quality+of+service

Lustre discussions:

• https://wiki.whamcloud.com/display/LNet/Lustre+QoS – see this!
• https://groups.google.com/g/lustre-discuss-list/c/n6sdj-e5LNA
• https://www.mail-archive.com/lustre-discuss@lists.lustre.org/msg04092.html
• https://admire-eurohpc.eu/wp-content/uploads/2023/12/Lustre-QoS-Barcelona-GA-Dec-2023.pptx.pdf

Intel Fabric Suite (configrms Lustre ServiceID)

• https://www.intel.com/content/dam/support/us/en/documents/network-and-i-o/fabric-products/Intel_OP_FabricSuite_Fabric_Manager_UG_H76468_v15_0.pdf

Any autonomous system must, by definition, take decisions. It is rational to consider the utility of the decision space and pick a nice point. We draw on our experience and simulate the impact of different decisions in our head, and then we pick one. Is that it?

No, right? Decisions are not irrevocable. You can adapt. If you accidentally oversped, you can apply brakes.

Predictive decisions rely on implicit or explicit modeling. Models are, in the best case, an nth-order approximation of a complex system, with ideally a decent predictive value and a low residual. Intuition, experience, and wisdom are also just models.

Models are by definition imperfect—a perfect model would also necessitate a perfect simulation of all reality. But they can be, and are, more or less imperfect. We all start off with more imperfect models, and (unless maladaptive) use the residuals from using their predictions to refine them. The decision loop, therefore, may be defined to have these components:

1. Observe inputs, and compute decision = f(input, intent)
2. Observe the impact of the decision and compute a course correction if necessary
3. Use the residual to update your hypothesis

Or a more formal attempt:

model: m_init   # Our initial model
intent: i_init  # Some mysterious missing ingredient

while Alive:
  reality = input()
  decision = f(reality, intent, model)
  impact = reality.apply(decision)
  error = intent - impact
  model.update(error)
  intent.update(reality, impact)

What Is An Optimal Decision?

It is tempting to then say that the quality of the initial decision does not matter, because it can always be updated. That models the system as a stochastic process, and is true insofar as that holds/is a reasonable approximation. But that is not the case: human lifetimes are finite and there is a time cost and material cost of bad decisions, there is no recovering from jumping off a cliff, prior actions also affect other agents’ models (what we call reputation or trust).

It may then be tempting to say that you want the best model before you take any decisions. But models are by definition updated by taking decisions. Scheduling of decisions is a useful escape hatch/degree of freedom: you may decide to read a book to have a better model before taking a decision with a bigger modeled impact. Scheduling is perhaps a meta-decision: choosing to explore (deferring the big decision) rather than exploit.

Deferring is useful insofar as the marginal improvement in your model outweighs the opportunity cost of delaying a decision. Two examples of instances where deferring is bad:

1. There exists no decision that meaningfully enhances the utility of the deferred decision
2. The opportunity cost of deferment is large

Connections With Standard Terminology I think open-loop and closed-loop control in control theory are not appropriate parallels. What I am describing is maybe an adaptive closed-loop system? But an autonomous, adaptive, completely programmable closed-loop system where decisions update both the model and the intent. Or just intelligent and conscious life?

Why Is This Useful

How to design systems. If a system requires some information that is, by definition, not available at bootstrap, adaptive mechanisms are necessary to solve the problem.

Autonomous intelligence, maybe? With current ML models, backpropagation is a progressive hypothesis refinement loop, but without an intent. And then model weights are frozen and they are tasked with prediction. But being static, they are not refined, unless we account for periodic batch refinement that is conducted with additional data gathered since the previous run.

Are models sufficiently adaptive? Is it possible that even with static weights, they learn to simulate memories etc. within those weights? What are the limits of computations that an algorithm running on top of a static machine can express, versus those of an algorithm that can mutate the machine itself?

Services in 3FS

• Cluster manager: highly available, manages membership, config etc (uses zookeeper?)
• Metadata service: uses FoundationDB
• Storage service (chunk store?)
• Client (two implementations: FUSE-based, and a more performant one.)

Metadata in 3FS

The Ongoing Metadata Debate

Whether we need object stores or filesystem semantics has been a topic of conversation recently². My own views so far:

• I buy the argument that data accesses during a training run do not need a namespace. Maybe the namespace can be batch-updated later on, if it gets in the way and does not provide the required performance.
• I feel that the usability argument still applies. At some point, you need to think about what does all the data mean, what to keep around, what to prune. There are a million reasons – legal, regulatory, financial, just basic IT maintenance that require metadata. Long-term, it is easy to think about things as a group or a hierarchy.

I think performant fileystem namespaces is doable. It may get needlessly hard if you tie yourself to a spec or a standard. That is because this is in flux, and we do not yet have unanimous agreement on what the standard should look like.

The 3FS Approach

They use a key-value model for their metadata, built on top of FoundationDB. Each key represents an inode.

FoundationDB, as an aside, is a very interesting database. It is like a distributed RocksDB with support for strict serializability in transactions. This is interesting to me — normally you think of KV stores as single-node storage backends for DBMSes to use. Transactions are implemented at a SQL level. FDB implements transactions at the KV level, but is supposed to be a lower-level database you layer on top of. (Aside: I wonder what the end-to-end principle says about the correct layering). It is like a persistent distributed software transactional memory, that tries to be lock-free, with optimistic and multi-version concurrency control.

• File inodes are INOD<inode_id>, mapped to ownership, permissions, times., and chunk information etc. for files. Interestingly, they encode inode_id in little-endian to spread inodes across FDB nodes. Directory inodes begin with DENT, have bidirectional links to parents to ensure no loops.needs to be and should be super-fast.
• fstat, lookup, listdir etc. invoke read-only txns. crate, link etc. invoke read-write txns.

State in 3FS

3FS metadata stores are stateless. Any operation is a self-contained transaction.

The metadata service does store file descriptors for files opened in write mode, but not in read mode. This helps with bootstrapping training jobs. This IMO is super important!! Any parallel job, when bootstrapping, needs to retrieve a large number of files in read-only mode. This should be super-fast, but is not so with Lustre. I vaguely recall someone mentioning that they had to hack on a cache on top of Lustre to push .so libraries to MPI ranks, and thinking that this feels like a fundamental flaw in what is supposed to be a parallel filesystem.

Question: how do you handle a write request for a file that is currently open in read-only mode? Ideally you need to track the readers and revoke their lease, but that requires maintaining state at the metadata layer? I think that’s done by tracking versions?

Chunk Store

This is relatively straightforward. It is per-node, and comes with a RocksDB instance for chunk metadata. Chunk metadata is cached in-memory. They have 11 chunk sizes from 64 KiB to 64 MiB. I don’t think this is a big deal but metadata shouldn’t be too much, even with 64KiB chunks, it is a little surprising that they went for this number of chunk sizes. But eh. Updates are CoW – old chunks remain valid until update completes.

Chain Replication

They use CRAQ (Chained Replication with Apportioned Queries) for replication. This is just a fancy way of saying that any replica can respond to a read request, with the catch being that you may get a stale version. A node may have a committed version of a block and a pending version. If so, it lets the requester know the existence of both versions, and the decision on whether to tolerate staleness or try again is left to the requester. Note that this is different from the case where a chain replica does not even know of the existence of an update – I don’t know if stale reads are tolerated in that case (or maybe version numbers are used to read a consistent snapshot).

Zero-copy in FUSE!?

They use FUSE for clients, so as to not deal with debugging kernel panics. They define an interface called USRBIO, for zero-copy interaction between the userspace and the FUSE layer. USRBIO is inspired by io-uring (or the Verbs API for that matter). The FUSE process manages Verbs-registered memory and submission/completion rings, and does the dispatch etc. These zero-copy ops are only used for the data path, and metadata ops still go through regular FUSE APIs.

Misc

• Codebase is mostly C++ with some Rust.
• They use P³ to verify their protocols. P is apparently a formal verification language for event-driven distributed services. Very cool!
• They seem to use flatbuffers for serdes and their own RDMA/RPC wrappers on top of the verbs interface. There’s a lot of code there and I just briefly skimmed through it.
• They heavily use folly, Facebook’s assorted library of C++ abstractions (including coroutines).

Epilogue

Not sure what to make of this. Writing a parallel filesystem is a massive undertaking. While metadata bottlenecks is a well-known problem in parallel filesystems, offloading it entirely to a DB like FoundationDB is an interesting choice. I have been trying to revisit how Colossus/Tectonic manage their metadata, and it is Bigtable and ZippyDB respectively. Ceph is pretty much the only open-source filesystem to support multiple metadata servers (Lustre is unwieldy in multiple ways). 3FS is optimized for RDMA fabrics over the data path, and makes subtle choices (like cheap metadata reads) to optimize for a large parallel job bootstrapping. I remember reading somewhere that it is designed for small random accesses, and I don’t really see why that is the case. I will end with the caveat that there are a lot of things to keep straight in distributed/parallel filesystem discussions, I don’t really work on these, and this post probably has a bunch of mistakes.

(Oh their design notes do say that they enable random access to training samples, I don’t see how they optimize for random accesses, maybe they mean that their system will transfer partial chunks and not do aggressive prefetching etc?)

References

Goal for this post: let us try justifying SQL. Why it exists, why is it in a specific form etc.

SQL vs Relational Algebra

I think SQL is made-up and non-essential. It is syntactic sugar over relational algebra. Other declarative interfaces may be more convenient abstractions — I personally prefer dataframes and chaining of expressions.

If you think of the query plan as a dataflow tree leading up to a root, SQL is essentially a traversal over that tree, like in-order, pre-order etc. The order SQL follows, specifically, may be called a Weird Arbitrary Traversal (or WAT). That the query planning infra evolved to reflect dataflow is not surprising, data movement is expensive, be it across functions or machines, and the more data you discard closer to the source the better off you are.

These pedantic distinctions are important, because they enable us to reason about

On the Universality of Relational Algebra?

Okay so why does relational algebra exist in the form it does?

As a lower bound on this argument, a query interface could just be say… C. You write a blob, machine runs blob, you get output.

We do declarative interfaces to make life easier given certain domain-specific constraints. In the relational world, it is:

1. Leveraging the relational/tabular data model.
2. Leveraging distributed cluster resources.
3. Leveraging fine-grained scheduling, concurrency etc.
4. Getting easy access to indexes etc.
5. Enabling reduced data movement by exposing primitives that capture data flow
6. Enabling query optimization by exposing primitives that make it easier to reason about ordering and statistics
7. Enable reasoning about partitioning and shuffling using key constraints and joins

Do Relational Operators Do These Things?

To a large extent, yes. That is why relational algebra endures. SQL just pretends to be inseparable from that and most people do not ask too many questions.

You always need escape hatches. That is why UDFs exist. But beyond a point, they hinder the query planning infra’s ability to reason about what the blob does.

Who Deserves To Be In The Club?

What operators should be in this club?

• Can we justify the existence of all existing operators?
• Are there new ones that should be there but are not?
• Under what models do we need radically different solutions?

Relational algebra checks out as it is essentially set theory. Given a table, you can slice it horizontally or vertically. Indexing helps with horizontal slicing. Vertical slicing is harder in row-based stores but column-based stores have solutions. Selection and projection map nicely to dataflow, indexing etc. Division seems to be the only relational operator not directly used in query plans.

Where Am I Getting At?

There exist data models (such as multidimensional array-based scientific data) that are not relational. Graphs are one example. Nothing here is really new.

The question I am trying to answer is: what is the set of ideas that lead you to the perfect operators for these domains? Do they exist? Are they in conflict with relational algebra? Or can they be members of the extended family?

Note to self: read referenced material ¹²³⁴

References

This post will explore the following thesis:

1. All networks and variants are tradeoffs.
2. If there are tradeoffs, there is no permanent winning. You are always tuning to chase the window.

Disclaimer: this could be the most obvious statement in the world. Or maybe deep within the symbolic mines of queuing and scheduling, someone has said something that disproves all of it. Or maybe that that statement is obvious but the arguments below are flawed. I have no idea — this blog does not come with any guarantees for rigor or scrutiny.

Context: Thinking About CBFC vs PFC

The thing in the back of my mind recently has been — why does everyone complain about PFC and RoCEv2, and why do they not complain about Infiniband and CBFC (Credit-based Flow Control). Why could RoCEv2 not adopt CBFC? What do you gain and what do you lose? Do you still need higher-level congestion control with CBFC?

CBFC Is Not A Magic Pill

An upper bound to CBFC goodness is much easier to establish than its precise extent.

CBFC is not a magic pill. It can still lead to credit loops, head-of-line blocking, buffer overflows, packet loss emanating from it, and so on. It is also amenable to tuning: routing, topologies, credit management schemes etc. to reduce the likelihood of bad things happening.

CBFC absolutely provides no theoretical guarantee (I’m not talking about a bounds-type argument: under X flows and Y buffer sizes you prove a Z\% upper bound on packet loss). Also not everyone means the same thing when they talk about CBFC — the Infiniband spec on this is not very prescriptive — my understanding is that it describes packet formats, and some basic mechanics, but still leaves a lot of room for vendors to do better.

Guarantees Are Expensive

You could provide a lot of guarantees by implementing a network as an all-to-all network of separate links. That is essentially what a crossbar is. But we want to optimize for cost. So we do tiers and routing and non-uniform bandwidth and all that stuff. You lose out on worst-case performance but that’s the only way to get systems to scale.

Turns out that this applies for everything — including tail latencies, jitter etc. The only way to guarantee that you will move a packet in a predictable timely manner is to explicitly carve out room for that packet on the path before sending it out. If you care about maintaining a high utilization or cost or scalability, this is bad.

Given a guarantee, you can always make an argument that “we can relax this guarantee a little and get a lot more performance in the common case.” This is statistical multiplexing or oversubscription or optimistic concurrency control or whatever. These are incredibly important — these thoughtful relaxations are the reason that systems work and can be built for the prices that’re built for. But it is always nice to be explicit about what you are losing in return.

Overloading Creates More Risks

As the goal of this exercise was to understand CBFC, let us compare it with sender-driven mechanisms. Sender-driven congestion control overloads certain signals to infer network state — RTT and packet loss seem to be two of them. This works until something happens that changes the meaning of these numbers — bufferbloat being one example. An advantage, in principle, of CBFC is that it is explicitly communicating actual network state, so theoretically, it should be robust to some such issues. I do not know if that is actually realized in practice.

This is in no way trashing sender-driven mechanisms. A good system is one that gets the job done — clean conscience and mathematical elegance are the domains of whiteboard hoggers. The end-to-end principle and this layered approach to TCP/IP has taken us a long long long way. Overloading those signals is the practical thing to do. (Also note that I’m conflating flow control with congestion control — CBFC is very much the former, but I guess it dictates and comes bundled with a different congestion control in IB with ECNs and injection throttling).

Expectations for Future Congestion Control

This post is a precursor to me trying to understand all the cool new things in datacenter fabrics — Omnipath, Slingshot, UltraEthernet, maybe Globally Scheduled Ethernet?

If these arguments hold, none of these systems’ mechanisms will be perfect. There will always be cases where Bad Things$^{TM}$ happen. Part of the argument is that that may be okay, and an evaluation of their properties must be holistic.

Some Degree Of Tuning Is Okay?

• I can not pick an outfit that works in all weathers. I am destined to keep tuning.
• Some tuning is worse than others. If I wore an outfit that had only one shoe, I would have to hop and keep swapping my shoe from one foot to the other.
• An outfit that covers the expected swings over one day is sufficient for an outfit.
• A bag that covers the expected swings over one week is sufficient for a short trip.
• A wardrobe that covers the expected swings over the year in Pittsburgh is sufficient for most other cases.

What are the takeaways for systems?

• We are destined to tune. We should plan for that.
• It would be nice if we understood systems in terms of their coverage of design space.
• Nothing in this discussion rules out the existence of systems that cover strictly more of the design space than others. Probably someone has and/or will prove that certain aspects are pareto-optimal. But it’s still a small sheet over massive legs.

Chasing The Global Optima

Network sharing and scheduling mechanisms are probably by far the most well-studied and deployed examples of a distributed system trying to construct/reach a global optimum, and both mathematical and practical arguments of the challenges that comes with.

All systems would benefit from better decisions. Better decisions are enabled by a better view of the system state. But constructing this system state may be prohibitively expensive. It may also be a moving target. It will definitely have some uncertainty bounds because physics. It may also be possible that explicitly constructing the system state is not necessary — we can apparently go quite a bit with individual actors acting on simple rules. (Something about game theory and cooperative games comes to mind but I’m already wayyyy past my depth here).

Step 1 in this process is understanding Infiniband. Hyperscalers seem to like Infiniband and RDMA, but they also like their IP addresses and routing tables. RoCE and ROCEv2 were attempts to bridge the gaps, but they turned out to have problems. At this point, I know the talking points everyone rehashes — PFC storms, HoL blocking, deadlocks… but I don’t really understand any of it.

From my understanding, all of the changes in RoCEv2 were necessitated because it is hard to get RDMA to work over lossy networks. (As an aside, it seems to me that there is no such thing as a truly lossless network, but you can build a pretty good illusion of one with two properties: 1. Reduce the likelihood of packet loss, and 2. Handle retransmissions transparently at some lower layer). I haven’t heard anyone complain about how flow control works in infiniband, so the key is maybe to understand it. This post is an attempt at that.

Credit-based Flow Control in Infiniband

This is all based off slides here: ⁵.

Virtual Lanes and Service Levels

Credits are maintained per-VL on each HCA. VLs/SLs are technically different but equivalent for now. They are priority classes — Infiniband supports up to 16. VL15 is reserved for subnet management traffic, while the others are available for regular data. The difference is that the application requests service levels, and the SL-VL mapping is a management decision handled presumably by the subnet manager.

Basic Numbers

• Each Flow Control Block is 64B. One 64B send requires one credit to be authorized.
• A VL will issue a maximum of 2048 credits, which translates to a 128KB receive buffer.

  CBFC Actual

  There are 3 main terms: ABR, FCCL, FCTBS.

• FCCL: the credit limit (the max point up to which the sender has been authorized to send).
• ABR: total blocks received at the receiver so far.
• FCTBS: total blocks sent.

FCTBS - ABR: blocks in transit on the wire. FCCL - FCTBS: remaining credits for the sender. A send will be permitted if the size is smaller than this limit.

Flow Control vs Congestion Control

Flow control makes infiniband largely and inherently lossless. Rare occasions that cause packet corruption etc. may require retransmissions. The fabric will do retransmissions for you if Reliable Connected was requested. CBFC kicks in regardless of whether you use RC or UC or UD. I haven’t looked into how retransmissions for RC are managed.

Infiniband also has congestion control on top of flow control. Why both need to exist is not entirely clear to me yet. What I do know is that IB employs some variant of ECN (Explicit Congestion Notification) to help detect congestion (FECN and BECN). I don’t know what the endpoints do in response to ECNs.

Questions For Myself

• Why is it hard to retrofit CBFC on to ethernet?
• Why don’t all datacenter fabrics use CBFC?
• How does CBFC compare to loss-based congestion control.

  References

AFS Cells

I get to have a home directory on two CMU cells – andrew.cmu.edu and ece.cmu.edu. I don’t know how to discover all the cells you have a home directory in.

ssh ankushj@unix.andrew.cmu.edu # give them your andrew password
$ pwd
/afs/andrew.cmu.edu/usr18/ankushj
$ cd /afs/ece.cmu.edu/usr/ankushj
$ ls
Permission denied
$ aklog ece.cmu.edu
$ ls
top_secret_infiniband_content.txt
$ tokens
<all the cells I'm authenticated for>

# random AFS commands
$ fs listaliases
$ fs listquota

Hosting on ECE

This is straightforward. Your public_html is available on users.ece.cmu.edu/~ankushj

ECE also respects .htaccess

$ ls -a /path/to/ece/homedir/public_html
.htaccess index.html

Hosting on Andrew

This is more confusing. Target address is https://www.andrew.cmu.edu/user/ankushj/.

$ ls -a /path/to/andrew/homedir/www
.htaccess index.html

1. Andrew does not use .htaccess.
2. index.html is not automatically available. You have to go to https://www.andrew.cmu.edu/server/publish.html, type your username, hit publish, and the data gets copied to some “real hosting destination”.
3. .unpublish is a thing¹

Redirects

On ECE, I can set up a HTTP-level redirect via .htaccess.

RedirectMatch 301 ^/$ https://my.real.website

I previously tried the following, but it would redirect to https://my.real.website~ankushj. I am pretty sure that ChatGPT is just giving me a stupid redirect command, but the version above works and the version below does not — that’ll have to do for now.

Redirect 301 / https://my.real.website

From Andrew, I had to set up a HTTP redirect.

<!DOCTYPE html>
<html>
  <head>
        <meta http-equiv="refresh" content="0; url=https//my.real.website">
            <title>Redirecting...</title>
  </head>
  <body>
        <p>If you are not redirected, <a href="https://my.real.website">click here</a>.</p>
  </body>
</html>

Troubleshooting

# Permissions
$ fs sa www system:anyuser rl

References

(gdb) bt
#0  0x00007fbaa4653c30 in ?? ()
#1  0x0000000000000000 in ?? ()

Since we have an intact $pc (which refers to %rip), we can figure out the instructions it was executing.

(gdb) disassemble $pc-20,$pc+20
Dump of assembler code from 0x7fbaa4653c1c to 0x7fbaa4653c44:
   0x00007fbaa4653c1c:  add    %al,(%rax)
   0x00007fbaa4653c1e:  jmp    0x7fbaa4653a51
   0x00007fbaa4653c23:  nopl   0x0(%rax,%rax,1)
   0x00007fbaa4653c28:  mov    0x98(%r12),%rax
=> 0x00007fbaa4653c30:  cmpb   $0x48,(%rax)
   0x00007fbaa4653c33:  jne    0x7fbaa4653b90
   0x00007fbaa4653c39:  movabs $0x50f0000000fc0c7,%rdx
   0x00007fbaa4653c43:  cmp    %rdx,0x1(%rax)
End of assembler dump.

Let us inspect the registers.

(gdb) info reg
rax            0x323338342034342e  3617296722238387246
rbx            0x7fff678ebb50      140734930795344
rcx            0x7fba82086120      140439022231840
rdx            0x1                 1
rsi            0x1                 1

Okay so our %rax was clearly a gibberish address, no wonder dereferencing it failed. Now the question is what source file/line was mapped to $pc. ChatGPT says that the following can work:

(gdb) list *$pc
<no output>
(gdb) info symbol $pc
No symbol matches $pc.

ChatGPT also says that we can also dereference addresses using these, but first we need to know what library is laid out in our memory, and at what offset. Noting these down for later.

$ addr2line -e /path/to/your/executable 0xADDRESS
$ objdump -d -S /path/to/your/executable

Some more useful information, saving for later.

(gdb) info frame 0
Stack frame at 0x7fff678ebaf8:
 rip = 0x7fbaa4653c30; saved rip = 0x0
 called by frame at 0x7fff678ebb00
 Arglist at 0x7fff678ebae8, args:
 Locals at 0x7fff678ebae8, Previous frame's sp is 0x7fff678ebaf8
 Saved registers:
  rip at 0x7fff678ebaf0
(gdb) info frame 1
Stack frame at 0x7fff678ebb00:
 rip = 0x0; saved rip = 0x0
 caller of frame at 0x7fff678ebaf8
 Arglist at 0x7fff678ebaf0, args:
 Locals at 0x7fff678ebaf0, Previous frame's sp is 0x7fff678ebb00
 Saved registers:
  rip at 0x7fff678ebaf8
(gdb) info frame 2
No frame at level 2.

Let’s look at shared memory mappings now.

(gdb) info shared
No shared libraries loaded at this time.
(gdb) info proc mappings
Mapped address spaces:

          Start Addr           End Addr       Size     Offset objfile
      0x5585a6034000     0x5585a604b000    0x17000        0x0 /some/bin
      0x5585a604b000     0x5585a64ef000   0x4a4000    0x17000 /some/bin
      0x5585a64ef000     0x5585a6582000    0x93000   0x4bb000 /some/bin
      0x7fba3ad28000     0x7fba3c000000  0x12d8000        0x0 /dev/shm/...
      0x7fba40d29000     0x7fba40f41000   0x218000        0x0 /dev/shm/...

Alright, getting somewhere. I have no idea why info shared failed but info proc mappings did not. We want to find a mapping around the address 0x00007fbaa4653c30.

Found something.

0x7fbaa4647000     0x7fbaa4659000    0x12000     0x3000 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1

The difference between the base address and our $pc is 0xcc30. Add the offset 0x3000 to get 0xfc30.

$ addr2line -e /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 0xfc30
??;0

Okay well thx.

$ objdump -d -S /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
...
fc30:       80 38 48                cmpb   $0x48,(%rax)
    fc33:       0f 85 57 ff ff ff       jne    fb90 <_Unwind_GetTextRelBase@@GCC_3.0+0xe40>
    fc39:       48 ba c7 c0 0f 00 00    movabs $0x50f0000000fc0c7,%rdx
    ...

Okay this wasn’t super useful. This is just some GCC unwinding utility function after a segfault. At this point, I just decide to examine the entire stack.

(gdb) x/128xg 0x7fff678eba00
...
0x7fff678eba00: 0x00007fba82086040      0x00007fbaa465000b
0x7fff678eba10: 0x000000000000002e      0x0000000000000000
0x7fff678eba20: 0x0000000000000000      0x0000000000000000
0x7fff678eba30: 0x00007fff678eced8      0xb741446eb7f0e800
0x7fff678eba40: 0x00007fff678eceb0      0x00007fff678ebb50
0x7fff678eba50: 0x00007fff678ebbf8      0x00007fff678ebb50
0x7fff678eba60: 0x00007fff678ebe00      0x323338342034342d
...

Some patterns start to emerge. All values starting with 0x7fff are pointers to things on the stack. Things in the range of 0x7fbaa.. are probably related to instructions. We can also see the junk value 0x3233 that was implicated in the segfault.

Two hours later …

My approach was to examine the stack visually, find pointers with prefixes that I knew to map to code I wore, and try and dereference them to get an idea of where my program was when it crashed.

This is doable, but it is not as straightforward as you might think. The why requires going into how ELF binaries/shared libraries are loaded in the memory.

1. There is a /proc/<pid>/map corresponding to info proc mappings that we saw earlier.
2. Each ELF file is divided into segments, which are further divided into sections. Mapping happens at the granularity of a segment.
3. The mapped segment will have a different offset than the on-disk segment. This may have something to do with alignment and/or ASLR requirements. But the segment sizes are also different for me, between what is reported by gdb, and what is shown by readelf/objdump.

As a result, I was unable to map symbol addresses from the core dump to symbols in libraries effectively. There is theoretically no reason why gdb should not be able to do this automatically, and it does, for more benign cases. But it does not seem to load the shared libraries for me for this particular crash.

Wait …

Okay, I ran gdb with this specific sequence, and suddenly it chose to load shared libraries.

$ gdb
(gdb) set auto-solib-add off # do not auto-load solibs
(gdb) set substitute-path /dev/shm /dev/null # something for shm maps
(gdb) set solib-search-path /path/to/lib
(gdb) file /path/to/my/binary
(gdb) target core /path/to/core-file
(gdb) info sharedlibrary
(gdb) info sharedlibrary
From                To                  Syms Read   Shared Object Library
0x00007fbaa4e98350  0x00007fbaa4eaccd1  No          /lib/libx.so 
0x00007fbaa4e21a00  0x00007fbaa4e72bc9  No          /lib/liby.so
...
(gdb) sharedlibrary /path/to/libmycode.so
Reading symbols from ...

I have no idea which of the above did the trick. Consider it a magic sequence of commands for now.

The game plan now is to go through the stack with x/64xg $pc and beyond to look for familiar addresses and try to resolve them via the symbol table. I tried a bunch of random symbols, and finally hit jackpot.

(gdb) info symbol 0x00005585a6470c80
Serialize[...] in section .text of /my/binary

It was a buffer overflow in a serialization routine.

Conclusions

The battle between you and a coy-acting core dump is a battle of wills. Do not blink.