ejabberd development
  1. ejabberd development
  2. EJAB-926

Drop c2s connection if socket send times out

    Details

      Description

      Quote of Anton Belyaev from the ejabberd mailing list thread: 100 mb per c2s connection (+patch)

      I've been investigating why ejabberd spends so much memory and found out that some connections consumed >100 mb.

      All the large connections were blocked on socket send operation. Recently EJAB-746 was fixed, introducing a 15 sec timeout on socket send. The problem has still remained for me, nevertheless. In ejabberd_c2s module results of socket send are not examined at all, so if client just does not read the socket, c2s process will block every 15 seconds on every message.

      Discussion: why results of socket send are not handled in ejabberd_c2s at all? Or such errors are handled somewhere else?

      The send timeout socket option in trunk helps if client does not read the socket. But, if the client receives messages more often than timeout (15 sec), the client's process heap will grow still.

      My application (component) sends a message to every user once 3 seconds. So, such client session will grow their message queue very fast. And ejabberd crashed with out-of-mem.

      To aid this I wrote a simple patch (attached), which drops session if socket send times out. The patch is against r1931 of trunk.

        Issue Links

          Activity

          Hide
          Badlop
          added a comment -

          This is the original patch by Anton Belyaev for c2s connection dropping on timeout. I just updated the line indentation.

          Show
          Badlop
          added a comment - This is the original patch by Anton Belyaev for c2s connection dropping on timeout. I just updated the line indentation.
          Hide
          ekhramtsov
          added a comment -

          I think there is a simplier way to do this. We can define send/2 as:

          send(Socket, Data) ->
              case gen_tcp:send(Socket, Data) of
                  ok -> ok;
                  _  -> exit(normal)
              end.
          

          This stops any gen_fsm/gen_server in the case of an error without a crash report.

          Show
          ekhramtsov
          added a comment - I think there is a simplier way to do this. We can define send/2 as: send(Socket, Data) -> case gen_tcp:send(Socket, Data) of ok -> ok; _ -> exit(normal) end. This stops any gen_fsm/gen_server in the case of an error without a crash report.
          Hide
          Mickaël Rémond
          added a comment -

          Let me try to finish my own patch in the coming days.

          Show
          Mickaël Rémond
          added a comment - Let me try to finish my own patch in the coming days.
          Hide
          Christophe Romain
          added a comment -

          reminder: only few hours left for commit before 2.1.0-rc1 release

          Show
          Christophe Romain
          added a comment - reminder: only few hours left for commit before 2.1.0-rc1 release
          Hide
          Badlop
          added a comment -

          Mickael committed the improvement in ejabberd trunk SVN r2671. I added a small check for OTP R13B in r2680. The improvement was merged into ejabberd road-to-exmpp SVN r2681. And the small check was merged in r2682.

          Show
          Badlop
          added a comment - Mickael committed the improvement in ejabberd trunk SVN r2671. I added a small check for OTP R13B in r2680. The improvement was merged into ejabberd road-to-exmpp SVN r2681. And the small check was merged in r2682.

            People

            • Votes:
              2 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:
                Days since last comment:
                4 years, 27 weeks, 3 days ago

                Issue deployment