1. 15 Jun, 2016 1 commit
    • Jon Paul Maloy's avatar
      tipc: add neighbor monitoring framework · 35c55c98
      Jon Paul Maloy authored
      TIPC based clusters are by default set up with full-mesh link
      connectivity between all nodes. Those links are expected to provide
      a short failure detection time, by default set to 1500 ms. Because
      of this, the background load for neighbor monitoring in an N-node
      cluster increases with a factor N on each node, while the overall
      monitoring traffic through the network infrastructure increases at
      a ~(N * (N - 1)) rate. Experience has shown that such clusters don't
      scale well beyond ~100 nodes unless we significantly increase failure
      discovery tolerance.
      This commit introduces a framework and an algorithm that drastically
      reduces this background load, while basically maintaining the original
      failure detection times across the whole cluster. Using this algorithm,
      background load will now grow at a rate of ~(2 * sqrt(N)) per node, and
      at ~(2 * N * sqrt(N)) in traffic overhead. As an example, each node will
      now have to actively monitor 38 neighbors in a 400-node cluster, instead
      of as before 399.
      This "Overlapping Ring Supervision Algorithm" is completely distributed
      and employs no centralized or coordinated state. It goes as follows:
      - Each node makes up a linearly ascending, circular list of all its N
        known neighbors, based on their TIPC node identity. This algorithm
        must be the same on all nodes.
      - The node then selects the next M = sqrt(N) - 1 nodes downstream from
        itself in the list, and chooses to actively monitor those. This is
        called its "local monitoring domain".
      - It creates a domain record describing the monitoring domain, and
        piggy-backs this in the data area of all neighbor monitoring messages
        (LINK_PROTOCOL/STATE) leaving that node. This means that all nodes in
        the cluster eventually (default within 400 ms) will learn about
        its monitoring domain.
      - Whenever a node discovers a change in its local domain, e.g., a node
        has been added or has gone down, it creates and sends out a new
        version of its node record to inform all neighbors about the change.
      - A node receiving a domain record from anybody outside its local domain
        matches this against its own list (which may not look the same), and
        chooses to not actively monitor those members of the received domain
        record that are also present in its own list. Instead, it relies on
        indications from the direct monitoring nodes if an indirectly
        monitored node has gone up or down. If a node is indicated lost, the
        receiving node temporarily activates its own direct monitoring towards
        that node in order to confirm, or not, that it is actually gone.
      - Since each node is actively monitoring sqrt(N) downstream neighbors,
        each node is also actively monitored by the same number of upstream
        neighbors. This means that all non-direct monitoring nodes normally
        will receive sqrt(N) indications that a node is gone.
      - A major drawback with ring monitoring is how it handles failures that
        cause massive network partitionings. If both a lost node and all its
        direct monitoring neighbors are inside the lost partition, the nodes in
        the remaining partition will never receive indications about the loss.
        To overcome this, each node also chooses to actively monitor some
        nodes outside its local domain. Those nodes are called remote domain
        "heads", and are selected in such a way that no node in the cluster
        will be more than two direct monitoring hops away. Because of this,
        each node, apart from monitoring the member of its local domain, will
        also typically monitor sqrt(N) remote head nodes.
      - As an optimization, local list status, domain status and domain
        records are marked with a generation number. This saves senders from
        unnecessarily conveying  unaltered domain records, and receivers from
        performing unneeded re-adaptations of their node monitoring list, such
        as re-assigning domain heads.
      - As a measure of caution we have added the possibility to disable the
        new algorithm through configuration. We do this by keeping a threshold
        value for the cluster size; a cluster that grows beyond this value
        will switch from full-mesh to ring monitoring, and vice versa when
        it shrinks below the value. This means that if the threshold is set to
        a value larger than any anticipated cluster size (default size is 32)
        the new algorithm is effectively disabled. A patch set for altering the
        threshold value and for listing the table contents will follow shortly.
      - This change is fully backwards compatible.
      Acked-by: default avatarYing Xue <ying.xue@windriver.com>
      Signed-off-by: default avatarJon Maloy <jon.maloy@ericsson.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
  2. 07 Apr, 2016 2 commits
    • Jon Paul Maloy's avatar
      tipc: stricter filtering of packets in bearer layer · 5b7066c3
      Jon Paul Maloy authored
      Resetting a bearer/interface, with the consequence of resetting all its
      pertaining links, is not an atomic action. This becomes particularly
      evident in very large clusters, where a lot of traffic may happen on the
      remaining links while we are busy shutting them down. In extreme cases,
      we may even see links being re-created and re-established before we are
      finished with the job.
      To solve this, we now introduce a solution where we temporarily detach
      the bearer from the interface when the bearer is reset. This inhibits
      all packet reception, while sending still is possible. For the latter,
      we use the fact that the device's user pointer now is zero to filter out
      which packets can be sent during this situation; i.e., outgoing RESET
      messages only.  This filtering serves to speed up the neighbors'
      detection of the loss event, and saves us from unnecessary probing.
      Acked-by: default avatarYing Xue <ying.xue@windriver.com>
      Signed-off-by: default avatarJon Maloy <jon.maloy@ericsson.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
    • Jon Paul Maloy's avatar
      tipc: eliminate buffer leak in bearer layer · 4e801fa1
      Jon Paul Maloy authored
      When enabling a bearer we create a 'neigbor discoverer' instance by
      calling the function tipc_disc_create() before the bearer is actually
      registered in the list of enabled bearers. Because of this, the very
      first discovery broadcast message, created by the mentioned function,
      is lost, since it cannot find any valid bearer to use. Furthermore,
      the used send function, tipc_bearer_xmit_skb() does not free the given
      buffer when it cannot find a  bearer, resulting in the leak of exactly
      one send buffer each time a bearer is enabled.
      This commit fixes this problem by introducing two changes:
      1) Instead of attemting to send the discovery message directly, we let
         tipc_disc_create() return the discovery buffer to the calling
         function, tipc_enable_bearer(), so that the latter can send it
         when the enabling sequence is finished.
      2) In tipc_bearer_xmit_skb(), as well as in the two other transmit
         functions at the bearer layer, we now free the indicated buffer or
         buffer chain when a valid bearer cannot be found.
      Acked-by: default avatarYing Xue <ying.xue@windriver.com>
      Signed-off-by: default avatarJon Maloy <jon.maloy@ericsson.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
  3. 07 Mar, 2016 1 commit
  4. 20 Nov, 2015 1 commit
  5. 24 Oct, 2015 5 commits
  6. 16 Oct, 2015 1 commit
    • Jon Paul Maloy's avatar
      tipc: send out RESET immediately when link goes down · 282b3a05
      Jon Paul Maloy authored
      When a link is taken down because of a node local event, such as
      disabling of a bearer or an interface, we currently leave it to the
      peer node to discover the broken communication. The default time for
      such failure discovery is 1.5-2 seconds.
      If we instead allow the terminating link endpoint to send out a RESET
      message at the moment it is reset, we can achieve the impression that
      both endpoints are going down instantly. Since this is a very common
      scenario, we find it worthwhile to make this small modification.
      Apart from letting the link produce the said message, we also have to
      ensure that the interface is able to transmit it before TIPC is
      detached. We do this by performing the disabling of a bearer in three
      1) Disable reception of TIPC packets from the interface in question.
      2) Take down the links, while allowing them so send out a RESET message.
      3) Disable transmission of TIPC packets on the interface.
      Apart from this, we now have to react on the NETDEV_GOING_DOWN event,
      instead of as currently the NEDEV_DOWN event, to ensure that such
      transmission is possible during the teardown phase.
      Signed-off-by: default avatarJon Maloy <jon.maloy@ericsson.com>
      Acked-by: default avatarYing Xue <ying.xue@windriver.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
  7. 31 Jul, 2015 1 commit
  8. 21 Jul, 2015 1 commit
    • Jon Paul Maloy's avatar
      tipc: make media xmit call outside node spinlock context · af9b028e
      Jon Paul Maloy authored
      Currently, message sending is performed through a deep call chain,
      where the node spinlock is grabbed and held during a significant
      part of the transmission time. This is clearly detrimental to
      overall throughput performance; it would be better if we could send
      the message after the spinlock has been released.
      In this commit, we do instead let the call revert on the stack after
      the buffer chain has been added to the transmission queue, whereafter
      clones of the buffers are transmitted to the device layer outside the
      spinlock scope.
      As a further step in our effort to separate the roles of the node
      and link entities we also move the function tipc_link_xmit() to
      node.c, and rename it to tipc_node_xmit().
      Reviewed-by: default avatarYing Xue <ying.xue@windriver.com>
      Signed-off-by: default avatarJon Maloy <jon.maloy@ericsson.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
  9. 14 May, 2015 1 commit
  10. 09 May, 2015 1 commit
  11. 29 Apr, 2015 1 commit
  12. 10 Mar, 2015 1 commit
    • Jon Paul Maloy's avatar
      tipc: ensure that idle links are deleted when a bearer is disabled · 169bf912
      Jon Paul Maloy authored
      commit afaa3f65
      (tipc: purge links when bearer is disabled) was an attempt to resolve
      a problem that turned out to have a more profound reason.
      When we disable a bearer, we delete all its pertaining links if
      there is no other bearer to perform failover to, or if the module
      is shutting down. In case there are dual bearers, we wait with
      deleting links until the failover procedure is finished.
      However, this misses the case when a link on the removed bearer
      was already down, so that there will be no failover procedure to
      finish the link delete. This causes confusion if a new bearer is
      added to replace the removed one, and also entails a small memory
      This commit takes the current state of the link into account when
      deciding when to delete it, and also reverses the above-mentioned
      Reviewed-by: default avatarErik Hugne <erik.hugne@ericsson.com>
      Signed-off-by: default avatarJon Maloy <jon.maloy@ericsson.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
  13. 06 Mar, 2015 1 commit
  14. 27 Feb, 2015 1 commit
  15. 09 Feb, 2015 6 commits
  16. 12 Jan, 2015 5 commits
  17. 24 Nov, 2014 1 commit
  18. 21 Nov, 2014 5 commits
  19. 14 May, 2014 1 commit
    • Jon Paul Maloy's avatar
      tipc: improve and extend media address conversion functions · 38504c28
      Jon Paul Maloy authored
      TIPC currently handles two media specific addresses: Ethernet MAC
      addresses and InfiniBand addresses. Those are kept in three different
      1) A "raw" format as obtained from the device. This format is known
         only by the media specific adapter code in eth_media.c and
      2) A "generic" internal format, in the form of struct tipc_media_addr,
         which can be referenced and passed around by the generic media-
         unaware code.
      3) A serialized version of the latter, to be conveyed in neighbor
         discovery messages.
      Conversion between the three formats can only be done by the media
      specific code, so we have function pointers for this purpose in
      struct tipc_media. Here, the media adapters can install their own
      conversion functions at startup.
      We now introduce a new such function, 'raw2addr()', whose purpose
      is to convert from format 1 to format 2 above. We also try to as far
      as possible uniform commenting, variable names and usage of these
      functions, with the purpose of making them more comprehensible.
      We can now also remove the function tipc_l2_media_addr_set(), whose
      job is done better by the new function.
      Finally, we expand the field for serialized addresses (format 3)
      in discovery messages from 20 to 32 bytes. This is permitted
      according to the spec, and reduces the risk of problems when we
      add new media in the future.
      Signed-off-by: default avatarJon Maloy <jon.maloy@ericsson.com>
      Reviewed-by: default avatarYing Xue <ying.xue@windriver.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
  20. 23 Apr, 2014 3 commits