Message Return Mechanism V2
Basic mechanism
1. Remove existing message return mechanism (Basic.Return command, 'mandatory' field of Basic.Publish, 'cluster-id' property of basic content)
2. Add 'reject-exchange' (short string) to Exchange.Declare and Queue.Declare (messages that are rejected are passed to this exchange)
Predefined simple behaviour
1. Add reject_key property to basic content class (short string)
2. Create new exchange type 'reject' with same semantics as direct, except that it matches on reject_key instead of routing_key
3. Provide predefined instance of 'reject' exchange type named 'amq.reject'
Is there a need for a separate reject exchange for a queue? I can see that rejection is a consumer activity, and the consumer is perhaps more interested in the queue than the exchange. However the message being rejected was published through a particular exchange and the publisher may also wish to handle rejections.
It seems simpler to me to just define a reject exchange for an exchange, not for a queue.
Semantics similar to the topic exchange might be better for the reject exchange type as it would allow wildcard matching which could be useful for catch-all bindings.
I'm still a little unconvinced on the need for a separate exchange type and the reject_key. As the publisher sets the reject_key, why not just allow them to override the reject-exchange at that level if required? We could then I suspect tackle most use-cases with the existing set of exchange types and without the need for a special header.
Some headers indicating (a) that a message was rejected or unroutable and (b) the reasons were discussed at the face-to-face. I can see value in those, but perhaps they could be put in the field table (with the reserved 'x-' prefix)?
Here's a proposal: If a message is to be rejected brokers and/or queues look for "x-reject-exchange" in the field table. If it is found then forward to that exchange. If not drop the message. No other changes needed.
Now supposing I actually want reject-key behavior as proposed above. Then I do the following
* Create an ordinary header exchange called "reject"
* Publishers create their personal queue and bind to "reject" exchange with "reject-key=mykey"
* Publishers send messages with "x-reject-exchange=reject, reject-key=mykey"
In other words the desired behaviour can be wired together using existing exchange types. I think all the other scenarios discussed on this list can also be wired together by the consumer given this single degree of flexibility. For example Gordon would like rejects to go to a topic exchange - no problem. The nice thing here is that the reject behavior is determined per message so the consumer can make individual rejected messages go literally anywhere that AMQ can take them.
We could put reject-exchange into basic.publish, I'm not too pushed either way.
I like that! A little bit less explicit, but it sets a good example for furture changes in having no impact on the wire format.
Gordon:
Separate exchange for queue is required to implement DLQs. Messages stored in queue in the moment the queue is deleted should be delivered to DLQ.
Making reject exchange use topic exhcnage matching algorithm: I haven't seen use case that would require such semantics, but maybe. Don't know.
Need for a separate exchange type and the reject_key: This is not meant as something you MUST use. Whole reject_key/reject exchange concept is meant only as something to make life easier for naive users. You can do as well without it.
Error codes in rejected message: I personally don't like idea of modifying the message while being routed. So far we don't do that. If you are interested whether message was rejected by exchange or queue, you can use separate reject exchange for exchage and queue.
Alan:
Imagine rejected message is sent to reject-exchange where it is rejected once more! Reject-exchange would send it to itself, thus creating infinte loop.
"Separate exchange for queue is required to implement DLQs. Messages stored in queue in the
moment the queue is deleted should be delivered to DLQ."
But why not just deliver them to the reject-exchange for the exchange to which the message was published? I don't see why a separate exchange is required.
"Error codes in rejected message: I personally don't like idea of modifying the message while being routed"
I do see your point on this. I'm not sure one way or the other. Lets leave it unless anyone comes up with a convincing case.
"Imagine rejected message is sent to reject-exchange where it is rejected once more! Reject-exchange would send it to itself, thus creating infinte loop."
That seems easy enough to prevent: if the reject-exchange is me and I don't have a matching binding, drop it.
"But why not just deliver them to the reject-exchange for the exchange to which the message was published? I don't see why a separate exchange is required."
The exchange may already not exist at the time the queue is deleted.
That seems easy enough to prevent: if the reject-exchange is me and I don't have a matching binding, drop it.
I haven't wanted to go into this again, hoped that would be able to convince you with non-philosophical arguments :), but ok. This is in fact a philosphical question. Protocol tries to separate sender of message from routing specifics. I.e. sender publishes a message and sets its fields but none of the data he have set does affect routing mechanism in any strictly given way. How message is routed is specified by wiring. Adding reject-exchange to Basic.Publish can be thought of as overriding primary exchange's routing mechanism _by sender_ - thus violating above principle.
Re: the exchange being deleted already, that is a good point. However there is no gurantee there anyway. I'd suggest it is an uncommon scenario and can be handled by the routing the message to the default amq.return (of whatever type that is).
Re: the philosophical point, the routing key seems to have as its primary purpose the routing mechanism along with the exchange, both of which are specified in publication. You are also proposing setting the reject_key to allow the publisher to influence the routing of the message once it is deemed undeliverable. I don't see a great difference in allowing the reject_exchange to be specified. However, I think Alan's suggestion is better anyway. Other than the infinite looping fear, is there any other objection to that?
Routing_key is _used_ for routing. However, it does not have any strictly given routing semantics as say "deliver message to queue specified in routing key". Same applies to reject_key. Thus, if we need to deliver messages somewhere else we can do that by simply rewiring without need to modify the sender application.
'reject-exchange' field in Basic.Publish (or in the headers field, doesn't matter) does have such strict routing semantics: "if message isn't matched by any binding, deliver it to such and such exchange". So if you want to change routing mechanism, rewiring doesn't do - you have to modify sender apllication.
The publish method contains an exchange to use for delivery, if you want to change that exchange then you need to reconfigure your sender (or maybe change the type of the exchange). It does not seem unreasonable to me for the same approach to be applied to the exchange to use for handling delivery failures. You still decouple the producer of the message from any consumer which is the ultimate goal. I think we have to agree to disagree.
Yes, it's philosophical discussion so we may never agree on how it should work. However, consider Robert's use case where message is delivered to primary service, it that one is not available, it is delivered to secondary service and if even that one is not available, it is delivered back to sender. This scenario is possible to implement with original proposal, but not when reject exchange is specified in the message.
Yes, you are right and it is a good point. Not having a chained structure (where each exchange can be linked to another for 'return' processing) does prevent a multi-stage approach to handling 'returns'.
I think that you will need an error code in the returned message. Take the following 2 use cases:
1) Publisher sends message then gets it back. You have no idea why. I guess you know there is a range of reasons:
Queue not declared, wrong routing key, no subscribers(if immediate), expired message(w/ ttl).
I could imagine a client app that the end user types in a value that is used as the routing key. If they get the value wrong then there is no way to tell them they made a mistake.
2) Later when security is introduced so publishers have to be authenticated to send to a particular queue:
In a middleware layer without error codes using the wrong routing key that happens to exist but to which the publisher has no rights could not be interpreted correctly.
The returned message may indicated that there is a configuration problem i.e. publisher has not been granted publish rights or that the routing key is wrong.
However the middleware would have to assume that it came back because it has expired/no subscribers and so may simply resend it which would waste a lot of bandwidth.
The AMQP documentation likes to compare itself to SMTP here they do modify the headers of messages as it passes through each MTA so the routing information can be used in processing.
Perhaps we don't want to expose the routing information to the consumers (of which the producer is also one) but knowledge about the processing could be useful to the producer to understand the reason for failure. Indeed other features such as billing and SPAM detection(perhaps this is really unlikely in the real world but it might be an issue) could benefit from routing information.