Details

    • Type: New Feature
    • Status: Closed
    • Priority: Low
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: ejabberd 14.05
    • Component/s: XEP Support
    • Labels:
      None

      Description

      It would be nice if ejabberd supported XEP-0198: Stream Management, for both c2s and s2s. I'd feel more secure knowing that stanzas will be resent if lost.

      Existing implementations:

      • Psi: Gsoc'10
      • Prosody: preliminary patch, not yet published

        Activity

        Hide
        imaginator Simon Tennant added a comment -

        This is a crucial feature for anyone working on mobile XMPP clients. We receive many reports of dropped messages and it's irritating for users to have no guarantee that their messages have arrived.

        Show
        imaginator Simon Tennant added a comment - This is a crucial feature for anyone working on mobile XMPP clients. We receive many reports of dropped messages and it's irritating for users to have no guarantee that their messages have arrived.
        Hide
        mremond@process-one.net Mickaël Rémond added a comment -

        We have actually design a better alternative protocol that we are currently testing on a real life deployment.
        We will publish the specification soon.

        Show
        mremond@process-one.net Mickaël Rémond added a comment - We have actually design a better alternative protocol that we are currently testing on a real life deployment. We will publish the specification soon.
        Hide
        miniman42 Greg Balmer added a comment -

        Great to see that someone is addressing this problem!

        This feature is critical to our usecase as we are finding many occasions where our mobile devices are simply not receiving sent messages or where our servers are not receiving messages sent by the mobile devices. We think this is due to the holes apparent in GPRS connectivity at cell boundaries or through inappropriately configured network timeouts.

        Is there any update as to the alternative protocol mentioned above? How is the testing going?

        Show
        miniman42 Greg Balmer added a comment - Great to see that someone is addressing this problem! This feature is critical to our usecase as we are finding many occasions where our mobile devices are simply not receiving sent messages or where our servers are not receiving messages sent by the mobile devices. We think this is due to the holes apparent in GPRS connectivity at cell boundaries or through inappropriately configured network timeouts. Is there any update as to the alternative protocol mentioned above? How is the testing going?
        Hide
        mremond@process-one.net mremond@process-one.net added a comment -

        Hello

        Testing is going well up to now.
        It is deployed in production for several projects on mobile and we have no message loss at all no matter the network conditions.

        Show
        mremond@process-one.net mremond@process-one.net added a comment - Hello Testing is going well up to now. It is deployed in production for several projects on mobile and we have no message loss at all no matter the network conditions.
        Hide
        ssoloid Dan Rissin added a comment -

        Stream management is crucial to our use case too. I very much look forward to hearing how the testing is going, and seeing the spec. BYW - I find it odd that this would have priority Low - I would think that losing messages would be kinda important for a messaging server.

        Show
        ssoloid Dan Rissin added a comment - Stream management is crucial to our use case too. I very much look forward to hearing how the testing is going, and seeing the spec. BYW - I find it odd that this would have priority Low - I would think that losing messages would be kinda important for a messaging server.
        Hide
        mremond@process-one.net Mickaël Rémond added a comment -

        XEP-0198 is low priority because it is not well done, for the following reasons:

        • it miss important concepts like replay. this is fondamental for
          reliability.
        • it mixes several different concept like message reliability and stream resuming.

        We have already developed and deployed in production another approach with great success. The reliability has been perfect so far with good level of information and granularity: No message lost at all even on mobile. This part is high priority.

        We need to clean up our spec to make a XEP and publish that as a first step.

        Show
        mremond@process-one.net Mickaël Rémond added a comment - XEP-0198 is low priority because it is not well done, for the following reasons: it miss important concepts like replay. this is fondamental for reliability. it mixes several different concept like message reliability and stream resuming. We have already developed and deployed in production another approach with great success. The reliability has been perfect so far with good level of information and granularity: No message lost at all even on mobile. This part is high priority. We need to clean up our spec to make a XEP and publish that as a first step.
        Hide
        rstml Rustam Aliyev added a comment -

        Is there any update on this feature? When can we expect some news? There's growing demand for reliability and at the moment ejabberd have no means to deal with it.

        Show
        rstml Rustam Aliyev added a comment - Is there any update on this feature? When can we expect some news? There's growing demand for reliability and at the moment ejabberd have no means to deal with it.
        Hide
        imaginator Simon Tennant added a comment -

        My reading of XEP-0198 is that replay is there.

        Why the need to create a new XEP rather than work with the authors of XEP-0198 to fix areas that are seen as deficient?

        Show
        imaginator Simon Tennant added a comment - My reading of XEP-0198 is that replay is there. Why the need to create a new XEP rather than work with the authors of XEP-0198 to fix areas that are seen as deficient?
        Hide
        nverite Nicolas Vérité (Inactive) added a comment -

        Hi Simon,

        The ack mechanism is not fine grained enough, intermediate entities maybe need to be identified as such, as well as there is a need for per-message or per-stanza acknowledgement.

        Re-reading the XEP, there is a too small reference to replay/resend when messages are not acknowledged, there is no serious such policiy, like the number and frequency of retries.

        until a stanza has been affirmed as handled by the receiver, that stanza is the responsibility of the sender (e.g., to resend it or generate an error if it is never affirmed as handled by the receiver).

        Additionnally, this XEP introduces a new first-child elements for the stream root element, which are enable, enabled, a, and r. This may not be wishable, since there are only three first-child element: iq, presence, and message. This is just a matter of syntax I believe.

        XEP-0184: Message Receipts is a better XEP as the acknowledgement feature is concerned, though again the replay policy is not addressed, and the intertermediate entities are not there neither.

        Show
        nverite Nicolas Vérité (Inactive) added a comment - Hi Simon, The ack mechanism is not fine grained enough, intermediate entities maybe need to be identified as such, as well as there is a need for per-message or per-stanza acknowledgement. Re-reading the XEP, there is a too small reference to replay/resend when messages are not acknowledged, there is no serious such policiy, like the number and frequency of retries. until a stanza has been affirmed as handled by the receiver, that stanza is the responsibility of the sender (e.g., to resend it or generate an error if it is never affirmed as handled by the receiver). Additionnally, this XEP introduces a new first-child elements for the stream root element, which are enable, enabled, a, and r. This may not be wishable, since there are only three first-child element: iq, presence, and message. This is just a matter of syntax I believe. XEP-0184: Message Receipts is a better XEP as the acknowledgement feature is concerned, though again the replay policy is not addressed, and the intertermediate entities are not there neither.
        Hide
        mremond@process-one.net Mickaël Rémond added a comment -

        We will publish our textone protocol subset specification before the end of the year.

        Show
        mremond@process-one.net Mickaël Rémond added a comment - We will publish our textone protocol subset specification before the end of the year.
        Hide
        imaginator Simon Tennant added a comment -

        XEP-0198 provides stream reliability. XEP-0184 provides message reliability. You use both together to achieve reliable delivery.

        There are already existing server and client implementations.

        I've not seen any discussion of your ACK-ing problems apart from here. Did you attempt to work with the authors of the existing specs to get ACK-ing problem addressed with the existing specs before coming up with something new and incompatible?

        Show
        imaginator Simon Tennant added a comment - XEP-0198 provides stream reliability. XEP-0184 provides message reliability. You use both together to achieve reliable delivery. There are already existing server and client implementations. I've not seen any discussion of your ACK-ing problems apart from here. Did you attempt to work with the authors of the existing specs to get ACK-ing problem addressed with the existing specs before coming up with something new and incompatible?
        Hide
        mremond@process-one.net Mickaël Rémond added a comment -

        XEP-0198 provides stream reliability. XEP-0184 provides message reliability. You use both together to achieve reliable delivery.

        In my understanding this is not enough. And as mention, stream management mix several things in a non elegant way.

        I've not seen any discussion of your ACK-ing problems apart from here. Did you attempt to work with the authors of the existing specs to get ACK-ing problem addressed with the existing specs before coming up with something new and incompatible?

        Yes, we mentionned inconsistencies years ago.
        We will publish a specification and the any one will be free to implement it if they want. It reused a large part of the same concept (like XEP-0184) but make it more useful, at least for our use case (which is mobile).

        Show
        mremond@process-one.net Mickaël Rémond added a comment - XEP-0198 provides stream reliability. XEP-0184 provides message reliability. You use both together to achieve reliable delivery. In my understanding this is not enough. And as mention, stream management mix several things in a non elegant way. I've not seen any discussion of your ACK-ing problems apart from here. Did you attempt to work with the authors of the existing specs to get ACK-ing problem addressed with the existing specs before coming up with something new and incompatible? Yes, we mentionned inconsistencies years ago. We will publish a specification and the any one will be free to implement it if they want. It reused a large part of the same concept (like XEP-0184) but make it more useful, at least for our use case (which is mobile).
        Hide
        imaginator Simon Tennant added a comment -

        I think it's great that you work on an implementation first and then write a spec based off what you learned from implementing. In my opinion there is not enough of that in the XSF.

        The realities are that Prosody and M-Link already implement XEP-0198. Ejabberd is implement something incompatible with the existing message ACK-ing approach.

        I urge you to re-have that discussion before writing yet-another-spec and we have an ejabberd-only standard and the lock-in that creates for client developers. I tried to find your original discussion on this. The closest I could find is 2006 (http://thread.gmane.org/gmane.network.jabber.standards-jig/13485/focus=13516). Is it worth bringing this up on Standards again?

        From a server admin and developer poing of view, I can't say I'm very impressed by the prospect of having two different standards that try to achieve the same thing. I would rather have one agreed upon standard that works across all servers.

        Show
        imaginator Simon Tennant added a comment - I think it's great that you work on an implementation first and then write a spec based off what you learned from implementing. In my opinion there is not enough of that in the XSF. The realities are that Prosody and M-Link already implement XEP-0198. Ejabberd is implement something incompatible with the existing message ACK-ing approach. I urge you to re-have that discussion before writing yet-another-spec and we have an ejabberd-only standard and the lock-in that creates for client developers. I tried to find your original discussion on this. The closest I could find is 2006 ( http://thread.gmane.org/gmane.network.jabber.standards-jig/13485/focus=13516 ). Is it worth bringing this up on Standards again? From a server admin and developer poing of view, I can't say I'm very impressed by the prospect of having two different standards that try to achieve the same thing. I would rather have one agreed upon standard that works across all servers.
        Hide
        mremond@process-one.net Mickaël Rémond added a comment -

        The problem is not with us. To speak with someone, there need to be someone willing to listen.
        So, we are open to discussion but in the current state we have no choice than to propose a better specifications and let interested implementors work with it.

        Show
        mremond@process-one.net Mickaël Rémond added a comment - The problem is not with us. To speak with someone, there need to be someone willing to listen. So, we are open to discussion but in the current state we have no choice than to propose a better specifications and let interested implementors work with it.
        Hide
        imaginator Simon Tennant added a comment -

        As far as I can tell, there has been no discussion so nobody can listen.

        Let's do this: we bring this to the standards list, you can describe the problems with the existing spec and we can work together to either scrap the existing XEPs or fix them. To me that seems a much better solution than having competing XEP and servers with different solutions to a common problem.

        Especially before you pour a whole lot of work into writing new specs.

        S.

        Show
        imaginator Simon Tennant added a comment - As far as I can tell, there has been no discussion so nobody can listen. Let's do this: we bring this to the standards list, you can describe the problems with the existing spec and we can work together to either scrap the existing XEPs or fix them. To me that seems a much better solution than having competing XEP and servers with different solutions to a common problem. Especially before you pour a whole lot of work into writing new specs. S.
        Hide
        mremond@process-one.net Mickaël Rémond added a comment -

        We (I) brought the discussion to the list and there have been no interest to change.
        We will publish our specifications and if the standard list has any interest for what we did they can discuss it adopt it, improve it, reject it. That's how things progress in a open world.

        Show
        mremond@process-one.net Mickaël Rémond added a comment - We (I) brought the discussion to the list and there have been no interest to change. We will publish our specifications and if the standard list has any interest for what we did they can discuss it adopt it, improve it, reject it. That's how things progress in a open world.
        Hide
        mremond@process-one.net Mickaël Rémond added a comment -

        By the way, you should come to sea beyond event in Paris. Topic will be realtime mobile collaboration and communication.
        We will work with developer coming from various companies on those topic and several companies will be presenting their achievement.
        I feel this is a good opportunity to progress further.

        Show
        mremond@process-one.net Mickaël Rémond added a comment - By the way, you should come to sea beyond event in Paris. Topic will be realtime mobile collaboration and communication. We will work with developer coming from various companies on those topic and several companies will be presenting their achievement. I feel this is a good opportunity to progress further.
        Show
        nverite Nicolas Vérité (Inactive) added a comment - For reciprocal reference: http://mail.jabber.org/pipermail/standards/2010-October/023928.html
        Hide
        william William Key added a comment -

        Are there still plans to release your XEP?

        Show
        william William Key added a comment - Are there still plans to release your XEP?
        Hide
        ssoloid Dan Rissin added a comment -

        "Mickaël Rémond added a comment - 27/Oct/10 02:38 PM
        We will publish our textone protocol subset specification before the end of the year."
        Have the textone protocol subset specs been published? I dug around a bit but could not find them.

        Show
        ssoloid Dan Rissin added a comment - "Mickaël Rémond added a comment - 27/Oct/10 02:38 PM We will publish our textone protocol subset specification before the end of the year." Have the textone protocol subset specs been published? I dug around a bit but could not find them.
        Hide
        nverite Nicolas Vérité (Inactive) added a comment -

        @Dan: "TextOne Protocol" is not released yet: we are in the process of writing a humanly-readable documentation. We hope te release it soon.

        Show
        nverite Nicolas Vérité (Inactive) added a comment - @Dan: "TextOne Protocol" is not released yet: we are in the process of writing a humanly-readable documentation. We hope te release it soon.
        Hide
        jvliwanag Jan Vincent Liwanag added a comment -

        Is there a documentation on the TextOne protocol available yet?

        Show
        jvliwanag Jan Vincent Liwanag added a comment - Is there a documentation on the TextOne protocol available yet?
        Hide
        ge0rg Georg Lukas added a comment -

        As you have repeatedly failed to maintain your own deadline for your new proposal, maybe you can at least have a look at the XEP-0198 implementation for ejabberd that can be found on github: https://github.com/xek/ejabberd

        If it is properly implemented, I am sure many people would appreciate you merging it into the main tree.

        Show
        ge0rg Georg Lukas added a comment - As you have repeatedly failed to maintain your own deadline for your new proposal, maybe you can at least have a look at the XEP-0198 implementation for ejabberd that can be found on github: https://github.com/xek/ejabberd If it is properly implemented, I am sure many people would appreciate you merging it into the main tree.
        Hide
        badlop Badlop added a comment -

        The only desktop client that I could find implementing that XEP is Gajim. That branch implements the protocol v0.5, while the latest XEP is v1.3.

        Anyway, the most important: does the branch work correctly for you?

        Show
        badlop Badlop added a comment - The only desktop client that I could find implementing that XEP is Gajim. That branch implements the protocol v0.5, while the latest XEP is v1.3. Anyway, the most important: does the branch work correctly for you?
        Show
        neustradamus Neustradamus added a comment - Any news about this support ? Gajim: http://gajim.org/ Beem: http://beem-project.com/issues/400 Yaxim: http://yaxim.org/ Swift http://swift.im/ Smack: http://issues.igniterealtime.org/browse/SMACK-333 a Psi fork: https://github.com/tfar/psi-soc2010 (and Iris fork: https://github.com/tfar/iris-gsoc2010 ) Pidgin: http://developer.pidgin.im/ticket/14252 Prosody : https://code.google.com/p/prosody-modules/wiki/mod_smacks Isode M-Link : http://www.isode.com/products/m-link.html
        Hide
        neustradamus Neustradamus added a comment -

        Any news about this?

        Show
        neustradamus Neustradamus added a comment - Any news about this?
        Hide
        jkonieczny Jacek Konieczny added a comment -

        Yeah… we need some protocol for message delivery reliability implemented. Not some mythical "textone" promised years ago, but something that has client support. Just a hop-to-hop solution would be enough for start. Like the XEP-0198.

        Show
        jkonieczny Jacek Konieczny added a comment - Yeah… we need some protocol for message delivery reliability implemented. Not some mythical "textone" promised years ago, but something that has client support. Just a hop-to-hop solution would be enough for start. Like the XEP-0198.
        Hide
        devurandom Dennis Schridde added a comment - - edited

        Any news or progress on this matter?

        Show
        devurandom Dennis Schridde added a comment - - edited Any news or progress on this matter?
        Hide
        count Andreas 'count' Kotes added a comment -

        Please provide an update.

        Show
        count Andreas 'count' Kotes added a comment - Please provide an update.
        Hide
        holger Holger Weiß added a comment -

        I have an initial patch. If you'd like to test it:

        https://github.com/weiss/ejabberd

        This is the raw diff:

        https://github.com/weiss/ejabberd/compare/master...xep-0198.diff

        The patch currently implements stanza ACKs but not stream resumption. I'll look into that soon.

        I'd be very interested in any kind of feedback.

        Show
        holger Holger Weiß added a comment - I have an initial patch. If you'd like to test it: https://github.com/weiss/ejabberd This is the raw diff: https://github.com/weiss/ejabberd/compare/master...xep-0198.diff The patch currently implements stanza ACKs but not stream resumption. I'll look into that soon. I'd be very interested in any kind of feedback.
        Hide
        holger Holger Weiß added a comment -

        I've now pushed code that adds stream resumption support into my repository, so the XEP-0198 support is now expected to be pretty much complete (except that it's limited to c2s connections so far).

        Show
        holger Holger Weiß added a comment - I've now pushed code that adds stream resumption support into my repository , so the XEP-0198 support is now expected to be pretty much complete (except that it's limited to c2s connections so far).
        Hide
        devurandom Dennis Schridde added a comment -

        Great! I'm looking forward to see this in deployment, soon.

        Are you also working on S2S support?

        Show
        devurandom Dennis Schridde added a comment - Great! I'm looking forward to see this in deployment, soon. Are you also working on S2S support?
        Hide
        holger Holger Weiß added a comment -

        I might have a go at s2s support, but I'd like to get the c2s code production-ready, first.

        If anyone interested in this feature could have a look at the code and/or test it, that would help a lot

        Show
        holger Holger Weiß added a comment - I might have a go at s2s support, but I'd like to get the c2s code production-ready, first. If anyone interested in this feature could have a look at the code and/or test it, that would help a lot
        Hide
        holger Holger Weiß added a comment -

        I created a pull request, so I guess any discussion on my actual patch should go there:

        https://github.com/processone/ejabberd/pull/166

        Show
        holger Holger Weiß added a comment - I created a pull request, so I guess any discussion on my actual patch should go there: https://github.com/processone/ejabberd/pull/166
        Hide
        neustradamus Neustradamus added a comment -

        Good job Holger!

        Show
        neustradamus Neustradamus added a comment - Good job Holger!

          People

          • Votes:
            26 Vote for this issue
            Watchers:
            22 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development