• Fred Lotter's avatar
    nfp: flower: cmsg rtnl locks can timeout reify messages · 28abe579
    Fred Lotter authored
    
    
    Flower control message replies are handled in different locations. The truly
    high priority replies are handled in the BH (tasklet) context, while the
    remaining replies are handled in a predefined Linux work queue. The work
    queue handler orders replies into high and low priority groups, and always
    start servicing the high priority replies within the received batch first.
    
    Reply Type:			Rtnl Lock:	Handler:
    
    CMSG_TYPE_PORT_MOD		no		BH tasklet (mtu)
    CMSG_TYPE_TUN_NEIGH		no		BH tasklet
    CMSG_TYPE_FLOW_STATS		no		BH tasklet
    CMSG_TYPE_PORT_REIFY		no		WQ high
    CMSG_TYPE_PORT_MOD		yes		WQ high (link/mtu)
    CMSG_TYPE_MERGE_HINT		yes		WQ low
    CMSG_TYPE_NO_NEIGH		no		WQ low
    CMSG_TYPE_ACTIVE_TUNS		no		WQ low
    CMSG_TYPE_QOS_STATS		no		WQ low
    CMSG_TYPE_LAG_CONFIG		no		WQ low
    
    A subset of control messages can block waiting for an rtnl lock (from both
    work queue priority groups). The rtnl lock is heavily contended for by
    external processes such as systemd-udevd, systemd-network and libvirtd,
    especially during netdev creation, such as when flower VFs and representors
    are instantiated.
    
    Kernel netlink instrumentation shows that external processes (such as
    systemd-udevd) often use successive rtnl_trylock() sequences, which can result
    in an rtnl_lock() blocked control message to starve for longer periods of time
    during rtnl lock contention, i.e. netdev creation.
    
    In the current design a single blocked control message will block the entire
    work queue (both priorities), and introduce a latency which is
    nondeterministic and dependent on system wide rtnl lock usage.
    
    In some extreme cases, one blocked control message at exactly the wrong time,
    just before the maximum number of VFs are instantiated, can block the work
    queue for long enough to prevent VF representor REIFY replies from getting
    handled in time for the 40ms timeout.
    
    The firmware will deliver the total maximum number of REIFY message replies in
    around 300us.
    
    Only REIFY and MTU update messages require replies within a timeout period (of
    40ms). The MTU-only updates are already done directly in the BH (tasklet)
    handler.
    
    Move the REIFY handler down into the BH (tasklet) in order to resolve timeouts
    caused by a blocked work queue waiting on rtnl locks.
    Signed-off-by: default avatarFred Lotter <frederik.lotter@netronome.com>
    Signed-off-by: default avatarSimon Horman <simon.horman@netronome.com>
    Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
    28abe579