Eulora's Communication Protocol, restated.

Sunday, 20 May, Year 10 d.Tr. | Author: Mircea Popescu

This is a continuation/refinement of the original version proposed on Diana's blog a month ago.

1. Overall Goals:

  • 1.1. All communications between clients and server to be encrypted.
  • 1.2. Clients to be able to receive from server any data they lack (including maps, skins, sound or video content etcetera), on demand.
  • 1.3. Clients to be able to choose and adjust both the level of security and their volume of communications with the server, as they will ultimately have to pay for the load that they generate.

2. Explicit Dependencies :

  • 2.1. Eucrypt for RSA with Keccak-based OAEP and Serpent symmetric ciphering.

3. Data Structures :

  • 3.0. Basic types :
    • char / int8 (1 byte) ;
    • int16 (2 byte) ;
    • int32 (4 byte) ;
    • int64 (8 byte) ;
    • floati (4 byte) ;
    • message (4096 bits) ;
    • rsa-message (1960 bitsii) ;
    • object (size of 104 bits : int32iii followed by 3 int16s representing positioniv followed by 3 int8s representing rotationv ) ;
    • text (size of n+n/256+1 bytes ; where the leading byte is the bytecount of the 2nd segment and the 2nd segment is the bytecount of the third segment).
  • 3.1. Serpent Key Set :
    • int8 (type ID, =1), followed by
    • int8 (count of keys in this set, n), followed by
    • n*(4*int64 + int32) (32 bytes each key followed by a 4 byte ID calculated as the keccak hash of the key itself), followed by
    • an int8 flag (LSB bit set -- keys to be used to talk to client ; MSB set -- key to be used to talk to server ; client-set MSB is ignored), followed by
    • int16 (message countvi), followed by
    • padding to multiple of message length.

  • 3.2. Serpent Keys Management :
    • int8 (type ID, =2), followed by
    • int8 (count of server keys requested), followed by
    • int8 (count of client keys requested), followed by
    • int8 (idvii of serpent key preferred for further inbound messages), followed by
    • int8 (count of burned keys in this message), followed by
    • n*int8 (id of burned key), followed by
    • int16 (message count), followed by
    • padding to RSA-message length.
  • 3.3. File Transfer :
    • int8 (type ID, =3), followed by
    • text (fully specifiedviii filename), followed by
    • text (content of file), followed by
    • int16 (message count), followed by
    • int32 (keccak hash of file content), followed by
    • padding to multiple of message length.
  • 3.4. File Request :
    • int8 (type ID, =4), followed by
    • text (fully specified filename), followed by
    • int16 (message count), followed by
    • padding to message length.
  • 3.5. Client Actionix :
    • int8 (type ID, =5), followed by
    • text (fully specified action, see section 5), followed by
    • int16 (message count).
  • 3.6. World Bulletinx :
    • int8 (type ID, =6), followed by
    • text, consisting of int32 (keccak hash of top level itemxi), followed by
    • int8 (count of objects), followed by
    • object list , followed by
    • int16 (message count).
  • 3.7. RSA Key :
    • int8 (type ID, =7), followed by
    • int8 (protocol version), followed by
    • int16 (subversion), followed by
    • int64 (keccak hash of client binary), followed by
    • text (RSA key in the e,N,comment republican format), followed by
    • int16 (message count), followed by
    • padding to 4* RSA-message lengthxii. Note that the server may issue this message to the client, after which the client must use the new server key for Serpent Keys Management (message type 2) as the server will discard type 2 messages sent to its public RSA key thereafter.
  • 3.8. Object Request :
    • int8 (type ID, =8), followed by
    • int8 (count of objects), followed by
    • n*int32 (hash of object), , followed by
    • int16 (message count).
  • 3.9. Object Info :
    • int8 (type ID, =9), followed by
    • int8 (count of objects), followed by
    • n times int32 (hash of object) and text (object properties, as per extant game structures, including art files needed and so onxiii), followed by
    • int16 (message count). The server will set the "target" of the player on the last object in the list.

4. Protocol Mechanics :

  • 4.0. All communications between server and client will consist of messages. These messages may be encrypted either via eucrypt.RSA or eucrypt.Serpent. All RSA-encryoted messages will be exactly 8`192 bits in length ; Serpent messages may be smaller than 4`096 bits, or an integer multiple of 4`096 bits.
  • 4.1. The server will attempt to decipher messages of sizes other than 16`384 bits it receives from known clientsxiv by using the current preferredxv Serpent key. Should the result pass sanity check, it will be acted upon accordingly ; otherwise it will be silently discarded.
  • 4.2. The server will attempt to decipher messages of sizes exactly equal to 16`384 bits received from known clients by using the current preferred Serpent keyxvi. Should the result pass sanity check, it will be acted upon accordingly. Should the result not pass sanity check or not originate from a known client, the server will attemptxvii to decrypt the message using its RSA key. Should the result consist of a type 7 message, the server will create a new user if it does not already exist, and issue a type 1 message in response, encrypted to the client RSA key. Otherwise the server will respond with 4096 bits of random data and silently discard any further messages from that IP for 1 hour.
  • 4.3. The server will issue type 1 messages encrypted to the corresponding client RSA key in response to any client messages for as long as it doesn't have a preferred client Serpent key set. The client is responsible for either maintaining or explicitly burning ~all~ of these, and will pay for them in any case.
  • 4.4. The server will issue type 6 packets in response to relevant type 5 packets received -- these can either signify the acceptance or the rejection of the client action, and the client must adjust its internal state accordingly.

5. Character Actions :

  • 5.0. Lock :
    • int8 (type ID, =0), followed by
    • int8 (count of objects), followed by
    • n* int32 (object ids). Defaults to currently targetted item.
  • 5.1. Make :
    • int8 (type ID, =1), followed by
    • int32 (object id, defaults to current target), followed by
    • int32 (object id, defaults to current recipe in mind), followed by
    • int32 (object id, defaults to current equipped tool), followed by
    • int8 (count of objects). The order for producing count items is entered into the queue (note that crafting only progresses if the player and/or hireling NPCs find themselves in certain situations re equipment, position, etc).
  • 5.2. Explore : int8 (type ID, =2). The character will attempt to find some resources.
  • 5.3. Exchange :
    • int8 (type ID, =3), followed by
    • int32 (object id, the other party), followed by
    • int32 (object id, the trade itselfxviii), followed by
    • int8 (count of objects), followed by
    • n* int32 (object ids) and int64 (object count), followed by
    • int8 (flag, set to 0x10 to lock a trade and to 0x0c to approve a trade previously locked by both players).
  • 5.4. Attack :
    • int8 (type ID, =4), followed by
    • int32 (object id, the other party), followed by
    • int32 (object id, the battle itself (server set, exactly in the way trade works). This not currently implemented, except player setting itself the bomb results in instadeath.
  • 5.5. Repair :
    • int8 (type ID, =5), followed by
    • int32 (object id, defaults to current target), followed by
    • int32 (object id, defaults to current equipped tool).
  • 5.6. Move :
    • int8 (type ID, =6), followed by
    • int32 (destination id, defaults to current target), followed by
    • int32 (slot id), followed by
    • int32 (object id, of the item being moved), followed by
    • int32 (quantity moved).
  • 5.7. Train :
    • int8 (type ID, =4), followed by
    • int32 (object id, the other party), followed by
    • int32 (object id, the train session itself (server set, exactly in the way trade and battle work). This not currently muchly implemented, except some NPCs train for money -- but will get greatly expanded asap.

Please leave your comments below.

———
  1. Floating point item deliberately not specified. []
  2. See TMSR-RSA OAEP padding for details. []
  3. Representing the identifying hash of the object in question. []
  4. Coordinates X, Y and Z in that order. Because the map goes from -500 to +500, the relationship between the given figure (GF) and map coordinates (MC) is GF / 65.535 - 500 = MC. []
  5. As a full rotation is 2 pi, the relationship between the given figure (GF) and object rotation (OR) is GF / 128 * pi = OR. []
  6. Each client and the server will keep a count of messages they sent each other. This value must be incremented on each subsequent message sent by no less than 1 and no more than 255. []
  7. Keys are maintained by both client and server in a ring buffer 256 elements long. Whenever new keys are received they are inserted in the buffer in the order they were received ; this procedure keeps passive indices in sync denoting all mutual keys on both server and client. []
  8. See the names discussion. []
  9. This is never issued by the server. []
  10. This is never issued by the client. []
  11. As discussed in comments, the world is a hierarchical structure of objects within objects. []
  12. This specifically excludes comments longer than 1560 bits (1960*4-8-8-16-64-16-24-6144). []
  13. The complete list of these is currently exposed by the extant client, but in any case we'll publish a complete schematic. []
  14. IPs associated with a RSA key. []
  15. Server key preferrence is per-client. []
  16. Either the first key it sent, or otherwise the key it specified via a type 2 message. []
  17. This opperation will be lower priority on the server side -- the server will more or less handle all waiting serpent messages before handling any RSA messages. []
  18. This is set by server through a type 6 message for both players involved, the trade is an object like any other that the OP has to request. The server will also expire trades, enforce them etc. []
Category: S.MG
Comments feed : RSS 2.0. Leave your own comment below, or send a trackback.

18 Responses

  1. Should there be a flag in 5.3 (and 5.4) for the client to initiate a new trade (or a new attack) where there exists no trade object yet?

    Should there be a character action to signal player movement (x y coords, direction, speed)?

    Should there be character actions for 'speaking' (say, tell, shout, etc.)?

  2. Mircea Popescu`s avatar
    2
    Mircea Popescu 
    Monday, 21 May 2018

    In order :

    Why ? Player aiming to initiate trade sends 5.3 ; server creates new object ; both players involved can review object and send further 5.3 packets, setting the flag to 10 or 0c as appropriate. Server destroys all trades that are older than some interval ; and enforces all ones that complete.

    There is : 5.6.

    Nah, that's all moving to irc ; client is expected to implement chat facilities (via #eulora) but is not specifically required to.

  3. In case of the text type, it consists of 1 byte (length of next block) + 255 text bytes. It's minimum length will be at least 256 bytes long. How can a shorter text be parsed? will it be zero padded?

    For message 7, the length calculation does not account for the byte in front of each segment. Also the comma's seem a bit strange, no text encoding of the e or N is specified (base64, hex or dec?). So these can contain comma's too.

    BTW in the calculation (1960*4-8-8-16-64-24-6144) which part contains the comma's and should the 24 not be 16 (for message count?)

  4. Mircea Popescu`s avatar
    4
    Mircea Popescu 
    Monday, 21 May 2018

    1. No, actually, the single byte text (say the character "C") will appear as 0x01 0x01 0x43, ie, one byte 2nd segment, one byte 3rd segment, byte "C". It's true you get a 200% overhead for this case, but then again text isn't really intended for very short fields. Were we encoding the longer string "ABC", it'd have read 0x01 0x03 0x041 0x42 0x43, 5 total bytes.

    2. It's not clear to me what you mean. For one thing, the e is always 2048 bits, and the N is always 4096 bits. The text field will therefore be always 0x02 followed by a value between 6144 (0x1800) and 7720 (0x1E28) reflecting the available space for the comment field. I suppose my statement is misleading with the commas comment -- I'm rephrasing that.

    3. The total available space for the message is 4 1960 bit chunks, which once OAEP'd and RSA'd result in 4 4096 bit chunks, which is the set size of the item. Out of this 4 * 1960 = 7840 bits, we use : 8 for the type id ; 8 more for the protocol version ; 16 more for the protocol subversion ; 64 for the client hash ; another 16 for the message count, and a set 6144 for the e and N. What's left behind is a maximum of 1960*4-8-8-16-64-16-24-6144 = 1560 (the 24 is because of the 3 bytes needed by the text header system).

  5. 2. Ah now I get my mistake with 1., There are 3 segments; the first for the length (in bytes) of the length of the text. The second for the length of the text. The last for the actual bytes. (I assumed the segments to be max 255 bytes length blocks, each block having the length in bytes of the next bytes)

  6. Anonimosu`s avatar
    6
    Anonimosuinsigna de prim sositinsigna de tehnolog 
    Monday, 21 May 2018

    If you fully specified it, by adding another byte in front to say how many segments are involved it would extend to perfection: could encode up to 256 segments, the first of which one byte long, the next up to 256 bytes long, the third up to 256^256 bytes long, the fourth uo to 256 ^ 256 ^ 256 (2.6e157826) bytes long. I can't even calculate what the theoretical maximum size of this structure would even come to.

    Speaking of which, why are you counting from 1 rather than from 0? It just costs you ~0.4% of the total addressable space for no clear benefit.

  7. Mircea Popescu`s avatar
    7
    Mircea Popescu 
    Tuesday, 22 May 2018

    @Ave1 Exactly.

    @Anonimosu Eh, there's really no need for that. But yes, in principle.

    Anyway, seems a very inconsquential loss ; and you get the benefit of more intuitive counters. Though, I suppose, the question of which style of counting is "more intuitive" isn't so easily resolved. I suppose it could be altered to count from 0, personally I don't feel too strongly about it either way.

  8. The slots vs coords issue starts annoying me beyond measure: why exactly should containers have "slots" with ids rather than coords inside them, anyway? Why have 'craft-table has 100 "slots" with ids from 0 to 99' instead of 'craft-table has internal space with length 20 and height 5, therefore available coordinates for stuff inside in the form (x,y) with x from 0 to 19 and y from 0 to 5'? Sure, the (x,y) is now the "slot id" if one wants to think of it that way, but I don't see the benefit of this "slot" differentiation really.

    Basically 5.6 (and Mocky's question above I suspect) illustrates this point: move as described by 5.6 expects a "slot id " which works fine when moving things *inside some container* but not so well when moving things *in the world* (i.e. changing their x,z coordinates). I'm not even sure how exactly is 5.6 supposed to reflect a change of x,z coordinates?

    Other questions:
    1. What is the meaning of "lock" for exchange? (not to mention, if it is lockable now, why not use 5.0 to lock it?)
    2. How does one cancel an ongoing thing (e.g. exchange, training session, craft-that-doesn-t-end-until-the-world-goes-out-in-flames)?
    3. Is the "top level" in 3.6 (world bulletin) meant as "id of item that contains this following list of stuff" (i.e. not necessarily top level in the full hierarchy as that would always be some world's id)?

  9. re: container slots vs. coords. I think the slots makes more sense, coords is more a function of how the client wants to display the container.

    re: 5.6, it doesn't seem compatible with player movement where coords are floats and also direction and speed are desirable to have.

  10. Mircea Popescu`s avatar
    10
    Mircea Popescu 
    Tuesday, 22 May 2018

    @Diana Coman I think you actually have a fine point, could just use coords, make them ints, be done with it.

    Re lock exchange : if exchanges are dual state, open and accepted, one player could steal some things from the box in the short interval before the other clicks to accept. So many years ago the three state was introduced, whereby a locked trade can no longer be modified, but only accepted, and an unlocked trade can be modified but not accepted, only locked.

    It could use 5.0, but I'd rather use the flag in here, not sure what the benefit of using 5.0 instead would be, seems just code bloat.

    One does not cancel anything. There is no cancel.

    Top level item is generally a sector (if outdoors) or a location (otherwise, like if in a room).

    @Mocky Speed is supposed to be enforced by client, server will just dump inacceptable movements reported.

  11. Dump bad movements, sure. But good speed and direction is not for the server to use but to pass to other clients.

  12. Mircea Popescu`s avatar
    12
    Mircea Popescu 
    Wednesday, 23 May 2018

    Why would server communicate speed/direction, I don't get it ?

  13. Currently there is this "movement modes" thing that client asks for and server replies with but it's just a legacy thing, no particular reason for it to exist. Basically server doesn't really care whether you "fly" or "run" or "jump on one foot" (or what such things even really mean from client's point of view) - all it cares is where you are at some given time.

  14. Right, the server doesn't have to care, and a text based client doesn't have to care. But a graphical client cares about the speed and direction of the *other* players in view.

    The client uses the speed of other players to choose between standing/walking/running/strafing animations for their avatars. And their direction to know if they are moving forward, backwards or sideways relative to themselves. Also to know if they are facing you or facing away or who/what they are fighting.

    Otherwise you get strange looking rendering of other players like they are moonwalking backwards or sliding around with no walking animation, fishing with back to the water, walking up to you and talking with back turned, all facing the same direction, etc.

  15. >> One does not cancel anything. There is no cancel.

    There has to be a cancel for trade. What would it mean to open a trade, add some items and then walk away? Trade goes on forever? Even after one player logs out? If not then how does the server notify the other player?

    If something happens where the trade can definitely not continue, the other player needs to know that fact.

    The server can detect if one party does something that will invalidate a pending exchange (e.g. moving out of range, logging out, zoning, dying, entering combat) and send a cancel to the other party in the world bulletin. However this should also be sent if one party closes the trade window, which requires a way for a client to send this signal to the server. One possibility would be a cancel flag on 5.3 that both client and server can use.

    I would suggest a more general solution, how does the server tell players that some item is no longer there (claim marker or bag of loot disappeared, player logged off)? perhaps instead of a cancel flag there could be a designated location where things move (5.6) to when they are to be considered gone.

    I also suggest to consider if an active exchange will block the parties from taking other actions or not. Can I trade while crafting, while in combat, while exploring, with multiple players at the same time? This impacts the protocol because if an active trade blocks other actions then you need to allow the second party to decline ever entering into the trade. If a player has the power to pop up a blocking trade window unbidden on my screen, he can disrupt me from crafting/exploring etc.

    If a trade window will be blocking, suggest to add an additional flag to 5.3: 'requested', or if not a flag some protocolitic way to accomplish the cancel without ever blocking the recipients crafting/exploring.

    Also, are items actually moved from inventory to exchange object in the same sense as a craft table where I can move heavy stuff to a trade window and continue with other activities/movement unburdened?

  16. Mircea Popescu`s avatar
    16
    Mircea Popescu 
    Wednesday, 23 May 2018

    > Right, the server doesn't have to care

    So if it cares let it figure it out ; as it's guess is exactly as good as the server and the server doesn't care to guess.

    > The client uses the speed of other players to choose

    Animations, see, are not really a server concern. I am aware ~nobody in "AAA games" is even aware, or would dare make the point ; but this makes entirely no difference. Items such as whether the OP is running or not are to be ~guessed~ by any client that cares to, not somehow be attempted at enforcement from the top down by the server ; and in the meta-sense, fuck the fucking barometer nonsense. We're doing virtual realities here, not spending our lives in a doomed effort to support the conceits of others at our own expense.

    The character object has a direction vector which tells you where it is facing. You can query it as often as you please, though it won't be free.

    > Otherwise you get strange

    If that's what the client wants to display, by all fucking means. This is a field where ~clients compete~ to deliver the best results ~the world permits~ to their players ; not a field where clients run over to mommy to "solve" their unsolvable problems for them (in the shape of presenting them with a certificate to show the problems were OfFiCiAlLy SoLvEd and honi soit qui mal y pense).

    See ?

  17. Mircea Popescu`s avatar
    17
    Mircea Popescu 
    Wednesday, 23 May 2018

    > What would it mean to open a trade, add some items and then walk away?

    The server keeps the trade object around for as long as it feels like, then reclaims it, and the objects drop to the ground. Where they stay until they are picked up or swept.

    Server does not notify ~anyone~. Other player can query for the trade object whenever feels like.

    > how does the server tell players that some item is no longer there

    By the clients asking for the top node and the server omitting it from the list.

    Inexistence is entirely passive, and not a positive thing.

    > I also suggest to consider if an active exchange will block

    The current blocking model is going away in favour of a queuing model for crafts ; in general blocking as now implemented is slated for retirement.

    > If a player has the power to pop up a blocking trade window unbidden

    This is a client concern. Don't like Windows-style clients, don't use.

    > Also, are items actually moved from inventory to exchange object

    I expect, yes.

  18. Mircea Popescu`s avatar
    18
    Mircea Popescu 
    Saturday, 26 May 2018

    On the basis of mature meditation and taking into consideration the commentary received, I've made the following changes :

    • The "object" object was redefined to include better coordinates (both position and rotation). Because of changes otherwise, this did not result in a significant bytesize increase.
    • The id hashes were universally reduced to 32 bits (with the exception of the client string -- mostly because that's not frequently communicated).

    Most modern machines will allign to bus size, meaning that 8 bit or 64 bit ints actually take the same space : 64 machine bits. While this is fine when the internals of a machine are involved, it is not at all fine on the internet ; clients will have to contain code to bytepack according to the above standard irrespective of their internal organization, so that packets actually take the specified number of bits.

Add your cents! »
    If this is your first comment, it will wait to be approved. This usually takes a few hours. Subsequent comments are not delayed.