Eulora's Communication Protocol, restated.

Sunday, 20 May, Year 10 d.Tr. | Author: Mircea Popescu

This is the second take (Oct 2nd) on a continuation/refinement of the original version proposed on Diana's blog a month ago. The original was published May 22nd. The author'd like to profusely thank the forum of The Most Serene Republic for invaluable help towards said refinement.

1. Overall Goals:

  • 1.1. All communications between clients and server to be encrypted.
  • 1.2. Clients to be able to receive from server any data they lack (including maps, skins, sound or video content etcetera), on demand.
  • 1.3. Clients to be able to choose and adjust both the level of security and their volume of communications with the server, as they will ultimately have to pay for the load that they generate.

2. Explicit Dependencies :

  • 2.1. Eucrypt for RSA with Keccak-based OAEP and Serpent symmetric ciphering.

3. Data Structures :

  • 3.0. Basic types :
    • char / int8 (1 byte) ;
    • int16 (2 byte) ;
    • int32 (4 byte) ;
    • int64 (8 byte) ;
    • floati (4 byte) ;
    • serpent-packet (1472 bytes) ;
    • rsa-messageii (1872 bitsiii) ;
    • rsa-packet (1470 bytes) ;
    • object (size of 104 bitsiv : int32v followed by 3 int16s representing positionvi followed by 3 int8s representing rotationvii ) ;
    • legacy-text (size of n+n/256+1 bytes ; where the leading byte is the bytecount of the 2nd segment and the 2nd segment is the bytecount of the third segment).viii
    • text (2 byte hearder containing the ~total~ byte length ; up to 1470 bytes of text ).

4. Serpent Packetsix :

  • 4.1. Serpent Key Set :
    • int8 (type ID, =1), followed by
    • int8 (count of keys in this set, n), followed by
    • n*(4*int64 + int32) (32 bytes each key followed by a 4 byte ID calculated through crc32x ), followed by
    • an int8 flag (LSB bit set -- keys to be used to talk to client ; MSB set -- key to be used to talk to server ; client-set MSB is ignored), followed by
    • int16 (message countxi), followed by
    • padding to Serpent-message length.

  • 4.2. Serpent Keys Management :
    • int8 (type ID, =2), followed by
    • int8 (count of server keys requested), followed by
    • int8 (count of client keys requested), followed by
    • int8 (idxii of serpent key preferred for further inbound Serpent-messages), followed by
    • int8 (count of burned keys in this message), followed by
    • n*int8 (id of burned key), followed by
    • int16 (message count), followed by
    • padding to Serpent-message length.
  • 4.3. File Transfer :
    • int8 (type ID, =3), followed by
    • text (fully specifiedxiii filename), followed by
    • text (content of file), followed by
    • int16 (message countxiv), followed by
    • padding to Serpent-message length.xv
  • 4.4. File Request :
    • int8 (type ID, =4), followed by
    • text (fully specified filenames, separated by ;xvi), followed by
    • int16 (message count), followed by
    • padding to Serpent-message length.
  • 4.5. Client Actionxvii :
    • int8 (type ID, =5), followed by
    • text (fully specified action, see section 5), followed by
    • int16 (message count), followed by
    • padding to Serpent-message length.
  • 4.6. World Bulletinxviii :
    • int8 (type ID, =6), followed by
    • int32 (id of top level itemxix), followed by
    • int8 (count of objects), followed by
    • object listxx, followed by
    • int16 (message count), followed by
    • padding to Serpent-message length.
  • 4.7. Object Request :
    • int8 (type ID, =7), followed by
    • int8 (count of objects), followed by
    • n*int32 (id of object), followed by
    • int16 (message count), followed by
    • padding to Serpent-message length.
  • 4.8. Object Info :
    • int8 (type ID, =8), followed by
    • int8 (count of objects), followed by
    • n times int32 (id of object) and text (object properties, as per extant game structures, including art files needed and so onxxi), followed by
    • int16 (message count), followed by
    • padding to Serpent-message length.
  • 5. RSA Packetsxxii :

    • 5.1. RSA key set.xxiii

      • int8 (equal to 251 to indicate packet contains a new RSA key), followed by
      • int8 (protocol version), followed by
      • int16 (subversion), followed by
      • int64 (keccak hash of client binary), followed by
      • int64 (e of RSA key), followed by
      • int8*490 (N of RSA key), followed by
      • int64 (preferred padding -- the magic value of 0x13370000 requests random padding ; all other values will be used as such, bitwise, ie like an infinite-length OTP consisting of the value repeated), followed by
      • int16 (message count), followed by
      • padding to RSA-message length, 1456 (5616-8-8-16-64-64-3920-64-16) bits exactly.
    • 5.2. Serpent key setxxiv :
      • int8 (equal to 157 to indicate packet contains new Serpent keys), followed by
      • int8 (count of keys in this set, n ; n<=40), followed by
      • n*(4*int64 + int32) (32 bytes each key followed by a 4 byte ID calculated as crc32 on the key itself), followed by
      • an int8 flag (LSB bit set -- keys to be used to talk to client ; MSB set -- key to be used to talk to server ; client-set MSB is ignored by server ; server will set LSB on keys requested by client for its own use thus supporting clients with no trustworthy random generators of their own), followed by
      • int16 (message count), followed by
      • padding to RSA-message length.

    • 5.3. Serpent Keys Management :
      • int8 (equal to 233 to indicate packet manages extant Serpent keys), followed by
      • int8 (count of server keys requested)xxv, followed by
      • int8 (count of client keys requested), followed by
      • int8 (id of serpent key preferred for further inbound Serpent-messages), followed by
      • int8 (count of burned keysxxvi in this message), followed by
      • n*int8 (id of burned key), followed by
      • int16 (message count), followed by
      • padding to RSA-message length.

    6. Protocol Mechanics :

    • 6.0. All communications between server and client will consist of messages. These messages may be encrypted either via eucrypt.RSA or eucrypt.Serpent. All RSA-encrypted messages will be exactly 1`470 bytes in length ; all Serpent messages will be exactly 1`472 bytes in length. The server will handle Serpent messages in preference of RSA messages (which are processed on an as-available basis). Clients that send garbage will be punished ; the costs involved (encryption/decryption ; generating entropy ; lookups and whatnots) will be pushed onto the client, for which reason writing the clients lightly pays off.
    • 6.3. The server will issue type 5.2 messages encrypted to the corresponding client RSA key in response to any client messages for as long as it doesn't have a preferred client Serpent key set. The client is responsible for either maintaining or explicitly burning ~all~ of these, and will pay for them in any case.
    • 6.4. The server will issue type 4.6 packets in response to relevant type 4.5 packets received -- these can either signify the acceptance or the rejection of the client action, and the client must adjust its internal state accordingly.

    7. Character Actions :

    • 7.0. Lock :
      • int8 (type ID, =0), followed by
      • int8 (count of objects), followed by
      • n* int32 (object ids). Defaults to currently targetted item.
    • 7.1. Make :
      • int8 (type ID, =1), followed by
      • int32 (object id, defaults to current target), followed by
      • int32 (object id, defaults to current recipe in mind), followed by
      • int32 (object id, defaults to current equipped tool), followed by
      • int8 (count of objects). The order for producing count items is entered into the queue (note that crafting only progresses if the player and/or hireling NPCs find themselves in certain situations re equipment, position, etc).
    • 7.2. Explore : int8 (type ID, =2). The character will attempt to find some resources.
    • 7.3. Exchange :
      • int8 (type ID, =3), followed by
      • int32 (object id, the other party), followed by
      • int32 (object id, the trade itselfxxvii), followed by
      • int8 (count of objects), followed by
      • n* int32 (object ids) and int64 (object count), followed by
      • int8 (flag, set to 0x10 to lock a trade and to 0x0c to approve a trade previously locked by both players).
    • 7.4. Attack :
      • int8 (type ID, =4), followed by
      • int32 (object id, the other party), followed by
      • int32 (object id, the battle itself (server set, exactly in the way trade works). This not currently implemented, except player setting itself the bomb results in instadeath.
    • 7.5. Repair :
      • int8 (type ID, =5), followed by
      • int32 (object id, defaults to current target), followed by
      • int32 (object id, defaults to current equipped tool).
    • 7.6. Move :
      • int8 (type ID, =6), followed by
      • int32 (destination id, defaults to current target), followed by
      • int32 (slot id), followed by
      • int32 (object id, of the item being moved), followed by
      • int32 (quantity moved).
    • 7.7. Train :
      • int8 (type ID, =4), followed by
      • int32 (object id, the other party), followed by
      • int32 (object id, the train session itself (server set, exactly in the way trade and battle work). This not currently muchly implemented, except some NPCs train for money -- but will get greatly expanded asap.

    Please leave your comments below.

    ———
    1. Floating point item deliberately not specified. []
    2. "Message" denotes the useful part ; "packet" denotes the hunk coming out of the interface. In the case of Serpent these are the same thing -- which should make it painfully obvious why we use it. []
    3. See TMSR-RSA OAEP padding for the principle and this discussion for details. []
    4. We really really want to keep this down. 13 bytes is the lowest I can conceive of, but I would so not mind halving it. []
    5. Representing the identifying hash of the object in question.

      We're using the narrower size to save on network traffic -- all the expenditure of another 32 bits here would buy us is de-ambiguation for cases where the count of objects around makes 1 in 2 billion collisions relevant. It doesn't seem likely a client could support such abundance of objects.

      Note that the hashes used here are client-specific, the server doesn't leak its own internal representation of objects to the clients. []

    6. Coordinates X, Y and Z in that order. Because the map goes from -500 to +500, the relationship between the given figure (GF) and map coordinates (MC) is GF / 65.535 - 500 = MC. []
    7. As a full rotation is 2 pi, the relationship between the given figure (GF) and object rotation (OR) is GF / 128 * pi = OR. []
    8. This arrangement permits the representation of arbitrarily large textfields (2nd segment can represent up to 115`792`089`237`316`195`423`570`985`008`687`907`853`269`984`665`640`564`039`457`584`007`913`129`639`936 bytes, which is more than enough space for all the text ever produced -- or likely to ever be produced -- by humanity) at the modest cost of a fixed 3 byte header.

      Unfortunately, it has no longer any utility for Eulora, since we've moved to fixed packets. I'm preserving it here because I really like it in the abstract and it has no other place to go. []

    9. These packets consist of 92 successive 128 bit chunks, Serpent-enciphered individually. To extract the payload one splits the message into 92 16-byte chunks, deciphers them the collates the output into a final result. To produce the packet one cuts a 11`776 bit payload into 92 128-bit chunks, Serpent-enciphers them, and collates the results into the outbound packet. []
    10. Generator polynomial 0x04c11db7. []
    11. Each client and the server will keep a count of messages they sent each other. This value must be incremented on each subsequent message sent by no less than 1 and no more than 255. []
    12. Keys are maintained by both client and server in an ordered ring buffer 256 elements long. Whenever new keys are received they are inserted in the buffer in the order they were received ; this procedure keeps passive indices in sync denoting all mutual keys on both server and client.

      The server will not sent more keys than'd fill either buffer, irrespective of request count. []

    13. See the names discussion. []
    14. If file is not chunked, message count will be 0. If file is chunked, message count will be 1 on first chunk, and 0 on last chunk. []
    15. Only if count=0 ; 4.3 packets with padding and message count != 0 are illegal. []
    16. Yeah, that's right. []
    17. This is never issued by the server. []
    18. This is never issued by the client. []
    19. As discussed in comments, the world is a hierarchical structure of objects within objects. []
    20. This portion will get more clarification later on! []
    21. The complete list of these is currently exposed by the extant client, but in any case we'll publish a complete schematic. The server will set the "target" of the player on the last object in the list. []
    22. These packets consist of three 490 byte successive chunks RSA-encrypted individually. To extract the payload one splits the message into three 490 byte chunks, RSA-decrypts and de-OAEP-pads each one, the collates the results into a final result. To produce the packet one cuts a 5`616 bit payload into three 1`872 bit chunks, OAEP-pads and encrypts them, and collates the results into the outbound packet. []
    23. This is the manner in which new clients register their RSA key with the server (thereby opening a new game account). Later replacement of a registered key IS NOT POSSIBLE. Keep your client's RSA key safe.

      Note that the server may also issue this message to any client, at any time but no more than once, after which the client must use the new server key for all subsequent RSA comms (the server will discard RSA messages sent to its public RSA key thereafter). []

    24. This permits either client or server to declare Serpent keys via RSA. It is not mandatory (as there exists a Serpent-encapsulated mechanism for the same end) but entirely legal. The server will always respond with at least one 5.2 packet after an accepted 5.1 packet creates a new player account, consisting of 40 Serpent keys to be used to talk to the server. Should the client respond with any other packet than 5.2 or 4.1, the server will send a 2nd 5.2 packet, containing 40 Serpent keys for the client's use. []
    25. Any values are permitted, server will sent multiple 5.2 packets in response if needed. []
    26. Client may only burn client keys ; server may only burn server keys. []
    27. This is set by server through a type 6 message for both players involved, the trade is an object like any other that the OP has to request. The server will also expire trades, enforce them etc. []
    Category: S.MG
    Comments feed : RSS 2.0. Leave your own comment below, or send a trackback.

    35 Responses

    1. Should there be a flag in 5.3 (and 5.4) for the client to initiate a new trade (or a new attack) where there exists no trade object yet?

      Should there be a character action to signal player movement (x y coords, direction, speed)?

      Should there be character actions for 'speaking' (say, tell, shout, etc.)?

    2. Mircea Popescu`s avatar
      2
      Mircea Popescu 
      Monday, 21 May 2018

      In order :

      Why ? Player aiming to initiate trade sends 5.3 ; server creates new object ; both players involved can review object and send further 5.3 packets, setting the flag to 10 or 0c as appropriate. Server destroys all trades that are older than some interval ; and enforces all ones that complete.

      There is : 5.6.

      Nah, that's all moving to irc ; client is expected to implement chat facilities (via #eulora) but is not specifically required to.

    3. In case of the text type, it consists of 1 byte (length of next block) + 255 text bytes. It's minimum length will be at least 256 bytes long. How can a shorter text be parsed? will it be zero padded?

      For message 7, the length calculation does not account for the byte in front of each segment. Also the comma's seem a bit strange, no text encoding of the e or N is specified (base64, hex or dec?). So these can contain comma's too.

      BTW in the calculation (1960*4-8-8-16-64-24-6144) which part contains the comma's and should the 24 not be 16 (for message count?)

    4. Mircea Popescu`s avatar
      4
      Mircea Popescu 
      Monday, 21 May 2018

      1. No, actually, the single byte text (say the character "C") will appear as 0x01 0x01 0x43, ie, one byte 2nd segment, one byte 3rd segment, byte "C". It's true you get a 200% overhead for this case, but then again text isn't really intended for very short fields. Were we encoding the longer string "ABC", it'd have read 0x01 0x03 0x041 0x42 0x43, 5 total bytes.

      2. It's not clear to me what you mean. For one thing, the e is always 2048 bits, and the N is always 4096 bits. The text field will therefore be always 0x02 followed by a value between 6144 (0x1800) and 7720 (0x1E28) reflecting the available space for the comment field. I suppose my statement is misleading with the commas comment -- I'm rephrasing that.

      3. The total available space for the message is 4 1960 bit chunks, which once OAEP'd and RSA'd result in 4 4096 bit chunks, which is the set size of the item. Out of this 4 * 1960 = 7840 bits, we use : 8 for the type id ; 8 more for the protocol version ; 16 more for the protocol subversion ; 64 for the client hash ; another 16 for the message count, and a set 6144 for the e and N. What's left behind is a maximum of 1960*4-8-8-16-64-16-24-6144 = 1560 (the 24 is because of the 3 bytes needed by the text header system).

    5. 2. Ah now I get my mistake with 1., There are 3 segments; the first for the length (in bytes) of the length of the text. The second for the length of the text. The last for the actual bytes. (I assumed the segments to be max 255 bytes length blocks, each block having the length in bytes of the next bytes)

    6. Anonimosu`s avatar
      6
      Anonimosuinsigna de prim sositinsigna de tehnolog 
      Monday, 21 May 2018

      If you fully specified it, by adding another byte in front to say how many segments are involved it would extend to perfection: could encode up to 256 segments, the first of which one byte long, the next up to 256 bytes long, the third up to 256^256 bytes long, the fourth uo to 256 ^ 256 ^ 256 (2.6e157826) bytes long. I can't even calculate what the theoretical maximum size of this structure would even come to.

      Speaking of which, why are you counting from 1 rather than from 0? It just costs you ~0.4% of the total addressable space for no clear benefit.

    7. Mircea Popescu`s avatar
      7
      Mircea Popescu 
      Tuesday, 22 May 2018

      @Ave1 Exactly.

      @Anonimosu Eh, there's really no need for that. But yes, in principle.

      Anyway, seems a very inconsquential loss ; and you get the benefit of more intuitive counters. Though, I suppose, the question of which style of counting is "more intuitive" isn't so easily resolved. I suppose it could be altered to count from 0, personally I don't feel too strongly about it either way.

    8. The slots vs coords issue starts annoying me beyond measure: why exactly should containers have "slots" with ids rather than coords inside them, anyway? Why have 'craft-table has 100 "slots" with ids from 0 to 99' instead of 'craft-table has internal space with length 20 and height 5, therefore available coordinates for stuff inside in the form (x,y) with x from 0 to 19 and y from 0 to 5'? Sure, the (x,y) is now the "slot id" if one wants to think of it that way, but I don't see the benefit of this "slot" differentiation really.

      Basically 5.6 (and Mocky's question above I suspect) illustrates this point: move as described by 5.6 expects a "slot id " which works fine when moving things *inside some container* but not so well when moving things *in the world* (i.e. changing their x,z coordinates). I'm not even sure how exactly is 5.6 supposed to reflect a change of x,z coordinates?

      Other questions:
      1. What is the meaning of "lock" for exchange? (not to mention, if it is lockable now, why not use 5.0 to lock it?)
      2. How does one cancel an ongoing thing (e.g. exchange, training session, craft-that-doesn-t-end-until-the-world-goes-out-in-flames)?
      3. Is the "top level" in 3.6 (world bulletin) meant as "id of item that contains this following list of stuff" (i.e. not necessarily top level in the full hierarchy as that would always be some world's id)?

    9. re: container slots vs. coords. I think the slots makes more sense, coords is more a function of how the client wants to display the container.

      re: 5.6, it doesn't seem compatible with player movement where coords are floats and also direction and speed are desirable to have.

    10. Mircea Popescu`s avatar
      10
      Mircea Popescu 
      Tuesday, 22 May 2018

      @Diana Coman I think you actually have a fine point, could just use coords, make them ints, be done with it.

      Re lock exchange : if exchanges are dual state, open and accepted, one player could steal some things from the box in the short interval before the other clicks to accept. So many years ago the three state was introduced, whereby a locked trade can no longer be modified, but only accepted, and an unlocked trade can be modified but not accepted, only locked.

      It could use 5.0, but I'd rather use the flag in here, not sure what the benefit of using 5.0 instead would be, seems just code bloat.

      One does not cancel anything. There is no cancel.

      Top level item is generally a sector (if outdoors) or a location (otherwise, like if in a room).

      @Mocky Speed is supposed to be enforced by client, server will just dump inacceptable movements reported.

    11. Dump bad movements, sure. But good speed and direction is not for the server to use but to pass to other clients.

    12. Mircea Popescu`s avatar
      12
      Mircea Popescu 
      Wednesday, 23 May 2018

      Why would server communicate speed/direction, I don't get it ?

    13. Currently there is this "movement modes" thing that client asks for and server replies with but it's just a legacy thing, no particular reason for it to exist. Basically server doesn't really care whether you "fly" or "run" or "jump on one foot" (or what such things even really mean from client's point of view) - all it cares is where you are at some given time.

    14. Right, the server doesn't have to care, and a text based client doesn't have to care. But a graphical client cares about the speed and direction of the *other* players in view.

      The client uses the speed of other players to choose between standing/walking/running/strafing animations for their avatars. And their direction to know if they are moving forward, backwards or sideways relative to themselves. Also to know if they are facing you or facing away or who/what they are fighting.

      Otherwise you get strange looking rendering of other players like they are moonwalking backwards or sliding around with no walking animation, fishing with back to the water, walking up to you and talking with back turned, all facing the same direction, etc.

    15. >> One does not cancel anything. There is no cancel.

      There has to be a cancel for trade. What would it mean to open a trade, add some items and then walk away? Trade goes on forever? Even after one player logs out? If not then how does the server notify the other player?

      If something happens where the trade can definitely not continue, the other player needs to know that fact.

      The server can detect if one party does something that will invalidate a pending exchange (e.g. moving out of range, logging out, zoning, dying, entering combat) and send a cancel to the other party in the world bulletin. However this should also be sent if one party closes the trade window, which requires a way for a client to send this signal to the server. One possibility would be a cancel flag on 5.3 that both client and server can use.

      I would suggest a more general solution, how does the server tell players that some item is no longer there (claim marker or bag of loot disappeared, player logged off)? perhaps instead of a cancel flag there could be a designated location where things move (5.6) to when they are to be considered gone.

      I also suggest to consider if an active exchange will block the parties from taking other actions or not. Can I trade while crafting, while in combat, while exploring, with multiple players at the same time? This impacts the protocol because if an active trade blocks other actions then you need to allow the second party to decline ever entering into the trade. If a player has the power to pop up a blocking trade window unbidden on my screen, he can disrupt me from crafting/exploring etc.

      If a trade window will be blocking, suggest to add an additional flag to 5.3: 'requested', or if not a flag some protocolitic way to accomplish the cancel without ever blocking the recipients crafting/exploring.

      Also, are items actually moved from inventory to exchange object in the same sense as a craft table where I can move heavy stuff to a trade window and continue with other activities/movement unburdened?

    16. Mircea Popescu`s avatar
      16
      Mircea Popescu 
      Wednesday, 23 May 2018

      > Right, the server doesn't have to care

      So if it cares let it figure it out ; as it's guess is exactly as good as the server and the server doesn't care to guess.

      > The client uses the speed of other players to choose

      Animations, see, are not really a server concern. I am aware ~nobody in "AAA games" is even aware, or would dare make the point ; but this makes entirely no difference. Items such as whether the OP is running or not are to be ~guessed~ by any client that cares to, not somehow be attempted at enforcement from the top down by the server ; and in the meta-sense, fuck the fucking barometer nonsense. We're doing virtual realities here, not spending our lives in a doomed effort to support the conceits of others at our own expense.

      The character object has a direction vector which tells you where it is facing. You can query it as often as you please, though it won't be free.

      > Otherwise you get strange

      If that's what the client wants to display, by all fucking means. This is a field where ~clients compete~ to deliver the best results ~the world permits~ to their players ; not a field where clients run over to mommy to "solve" their unsolvable problems for them (in the shape of presenting them with a certificate to show the problems were OfFiCiAlLy SoLvEd and honi soit qui mal y pense).

      See ?

    17. Mircea Popescu`s avatar
      17
      Mircea Popescu 
      Wednesday, 23 May 2018

      > What would it mean to open a trade, add some items and then walk away?

      The server keeps the trade object around for as long as it feels like, then reclaims it, and the objects drop to the ground. Where they stay until they are picked up or swept.

      Server does not notify ~anyone~. Other player can query for the trade object whenever feels like.

      > how does the server tell players that some item is no longer there

      By the clients asking for the top node and the server omitting it from the list.

      Inexistence is entirely passive, and not a positive thing.

      > I also suggest to consider if an active exchange will block

      The current blocking model is going away in favour of a queuing model for crafts ; in general blocking as now implemented is slated for retirement.

      > If a player has the power to pop up a blocking trade window unbidden

      This is a client concern. Don't like Windows-style clients, don't use.

      > Also, are items actually moved from inventory to exchange object

      I expect, yes.

    18. Mircea Popescu`s avatar
      18
      Mircea Popescu 
      Saturday, 26 May 2018

      On the basis of mature meditation and taking into consideration the commentary received, I've made the following changes :

      • The "object" object was redefined to include better coordinates (both position and rotation). Because of changes otherwise, this did not result in a significant bytesize increase.
      • The id hashes were universally reduced to 32 bits (with the exception of the client string -- mostly because that's not frequently communicated).

      Most modern machines will allign to bus size, meaning that 8 bit or 64 bit ints actually take the same space : 64 machine bits. While this is fine when the internals of a machine are involved, it is not at all fine on the internet ; clients will have to contain code to bytepack according to the above standard irrespective of their internal organization, so that packets actually take the specified number of bits.

    19. Where do the 8192 bits in 4.0, 16384 bits in 4.1 and in 4.2 come from?

      In other words: there's something I don't get regarding the sizes of messages at 4.0 and 4.1, as they seem to keep doubling and then doubling again. Why 8192 bits for a RSA message since a TMSR RSA-encrypted message is actually 4096 bits? This 4096 already includes the "doubling+" caused by OAEP (i.e. the RSA-payload transported in a message of 4096 bits is 1960 bits and that's fixed, as per http://ossasepia.com/2018/02/15/eucrypt-chapter-10-oaep-with-keccak-a-la-tmsr/#selection-63.0-65.271 ).

      As to Serpent messages, in principle they can be any size that is however a multiple of 128 bits since that's the length of a Serpent output block. So at 4.0 to 4.2 I'd just say that server will first attempt to decrypt using current Serpent key for that client and if that fails, it looks at message size: if it's precisely 4096 bits, then it will try decrypting with server's own RSA key and proceed according to result; if any other size, then discard.

    20. Mircea Popescu`s avatar
      20
      Mircea Popescu 
      Tuesday, 2 October 2018

      Yeah, I suspect the numbers there need rewriting, I spend a while trying to figure out wtf I was talkign about back in May but it's not coming back to me.

      So basically, the sane thing here is for the server to
      a) discard all messages whose size is not a multiple of 128 bits
      b) attempt to de-serpent all messages whose size is a multiple of 128 bits
      c) should b fail, and should the message size be exactly 4096 bits, server should attempt to de-rsa it.

      That right ?

    21. Sounds right to me, yes.

      At any rate, if there is some good reason for other numbers/different approach there I think it will become clear as soon as we get some prototype out so we'll see.

    22. Of course now I finally see what I just couldn't figure out earlier despite all the reading and re-reading before even asking the question: the 4096 bits size and specific 4.2 description is to allow the sending of RSA keys themselves (either by client or by server) via a 3.7 message that is padded to 4*4096, yes.

      The confusion arose from this contradiction between "RSA encrypted means 4096 size, fixed" on one hand and "sending RSA key in a RSA-encrypted 4*4096 message" on the other. Seeing the int16 message count in 3.7 there I tend to think that the idea here was that a 3.7 message will in fact be split into 4 parts with their corresponding counts so at message level it still IS "either 4096 bits or discard" (although the server will then have to keep track of parts from each IP for a while in this case) but I can't locate any proper discussion of this. So what's the approach for sending 3.7?

    23. I think I finally got it: the 3.7 there is meant to be simply 4 RSA-encrypted chunks put together in which case 4*4096 makes sense for total size of message and the "padding to 4*RSA-message length" in fact refers to 4*1960 (matches the footnote there too).

      I guess that "RSA-message" confused me as it's more of a RSA-payload basically, while the "message" being 4096 is a RSA-encrypted message.

    24. Mircea Popescu`s avatar
      24
      Mircea Popescu 
      Tuesday, 2 October 2018

      Yeah, the "message" as in, payload, and the "message" as in, encapsulated payload ambiguity is counterproductive. Though I did attempt to call the former a message and the latter a packet throughout, I guess I didn't do such a good job of sticking to it.

      Ima give the whole thing a re-read today and update it.

    25. " In the case of Serpent these are the same thing -- which should make it painfully obvious why we use it. " - uhm, not quite the same thing, merely the same size! From pov of protocol, same data structure though, indeed.

      "http://trilema.com/2018/euloras-communication-protocol-restated/#selection-1099.361-1099.513" - is this 2nd packet with *new* Serpent keys or same keys as before? Also, is this 2nd packet the last or does the server keep re-sending the same Serpent keys in reply to any client message that is not 5.2 or 4.1 until it receives one of those?

      http://trilema.com/2018/euloras-communication-protocol-restated/#selection-864.10-989.92 - I assume that keys are entered in the order they are generated (&sent) rather than received, since with UDP they might arrive in a different order so the only reliable indicator would be presumably the message count. What happens though if/when some keys don't make it at all to the client? The way I see it now, client would have to either ask the server to "burn" those it did not receive (if it's not the very first set so it still has some working Serpent key it can use to send the 3.2 management message) or otherwise, if it is the first set to just re-send the message so that the server in turn re-sends the keys. Is this the intended behaviour or am I missing some other option there?

      At 5.2 and 5.3 the id is on 8 bits so the magic values there should be much smaller as asciilifeform pointed out already in the log (http://btcbase.org/log/2018-10-02#1857434 )

    26. Mircea Popescu`s avatar
      26
      Mircea Popescu 
      Wednesday, 3 October 2018

      > uhm, not quite the same thing

      Right, I just meant it has no overhead. RSA message is Is this 2nd packet with *new* Serpent keys

      Server sends 40 Serpent keys for client to use to talk to it in any case. If client doesn't send Serpent keys for server to use to talk to it, then server sends client a further 40 Serpent keys, for server to talk to client.

      > I assume that keys are entered in the order they are generated

      There's something weird going on with the selection there. But yes, message count orders items irrespective of transport layer.

      Keys are burned one at a time ; and client selects whichth key the server to use anyway. There is no re-sending of already sent keys in any case.

      Good point re magix huh, 8 bits not 8 bytes. Will fix.

    27. Mircea Popescu`s avatar
      27
      Mircea Popescu 
      Thursday, 4 October 2018

      Further updates :
      - correct magic packet ids for the RSA packets ;
      - replaced a spurious keccak with a lighter (and more useful) crc32
      - removed a spurious keccak altogether (in 4.6).

    28. How about the ID in 5.2 ?

    29. Mircea Popescu`s avatar
      29
      Mircea Popescu 
      Thursday, 4 October 2018

      That's a server-computed item. The server creates unique per-client ids out of its own unique global ids via keccak.

    30. At 4.7 and 4.8 the n*int32 should also simply be "id of object."

      At 5.2 - what will server do with messages that have the count of keys set to anything above 40? Consider n=40 or discard the whole message as invalid?

    31. Mircea Popescu`s avatar
      31
      Mircea Popescu 
      Friday, 5 October 2018

      Updated 4.7 4.8

      All illegal packets (including a packet purporting to carry more keys than possible) should be a) discarded and b) marked, because clients sending illegal packets will be examined (so the author can debug) and/or punished (so as to protect the server).

    1. [...] This is a work in progress towards an Ada implementation of the communication protocol discussed on Trilema four months ago. [...]

    2. [...] draft Comms Protocol has improved markedly. Botted crafting now works pretty well also! ———This was same test server for [...]

    3. [...] reflects the fact that ALL messages are either 14721 or 14702 octets long in the newest version of Eulora's communication protocol. Here's the summary of the data I [...]

    4. [...] contained in any message has to be structured according to the protocol's rules described in sections 4, 5 and 7 of the specification. This layer provides the definitions of those data structures as well as the conversion methods [...]

    Add your cents! »
      If this is your first comment, it will wait to be approved. This usually takes a few hours. Subsequent comments are not delayed.