May 20, 2018 | Author: Mircea Popescu

This is the current take on an ongoing effort towards specification, last revised January 21st, 2021.

1. Overall Goals:

  • 1.1. All communications between clients and server to be encrypted.
  • 1.2. Clients to be able to receive from server any data they lack (including maps, skins, sound or video content etcetera), on demand.
  • 1.3. Clients to be able to choose and adjust both the level of security and their volume of communications with the server, as they will ultimately have to pay for the load that they generate.

2. Explicit Dependencies :

  • 2.1. Eucrypt for RSA with Keccak-based OAEP and Serpent symmetric ciphering.

3. Data Structures :

  • 3.0. Basic types :
    • char / uint8 (1 byte) ;
    • uint16 (2 byte) ;
    • uint32 (4 byte) ;
    • uint64 (8 byte) ;
    • floati (4 byte) ;
  • 3.1. Special types :
    • hash (128 bits) ;
    • chunk [of file] (bitfield, 11760 bits) ;
    • serpent-packet (1472 bytes) ;
    • rsa-messageii (1872 bitsiii) ;
    • rsa-packetiv (1470 bytes) ;
    • object (size of 104 bitsv : uint32vi followed by 3 uint16s representing positionvii followed by 3 uint8s representing rotationviii ) ;
    • legacy-text (size of n+n/256+1 bytes ; where the leading byte is the bytecount of the 2nd segment and the 2nd segment is the bytecount of the third segment).ix
    • text (2 byte hearder containing the ~total~ byte length ; up to 1470 bytes of text ).

4. Serpent Packetsx :

  • 4.1. Serpent Key Set :
    • uint8 (type ID, =100), followed by
    • uint8 (count of keys in this set, n), followed by
    • n*(4*int64 + uint32) (32 bytes each key followed by a 4 byte ID calculated through crc32xi ), followed by
    • an uint8 flag (LSB bit set -- keys to be used to talk to client ; MSB set -- key to be used to talk to server ; client-set MSB is ignored), followed by
    • uint16 (message countxii), followed by
    • padding to Serpent-message length.
  • 4.2. Serpent Keys Lifecycle Management :
    • uint8 (type ID, =102), followed by
    • uint8 (count of server keys requested), followed by
    • uint8 (count of client keys requested), followed by
    • uint8 (idxiii of serpent key preferred for further inbound Serpent-messages), followed by
    • uint8 (count of burned keys in this message), followed by
    • n*int8 (id of burned key), followed by
    • uint16 (message count), followed by
    • padding to Serpent-message length.
  • 4.4.a. File Request, manifest
    • uint8 (type ID, =3), followed by
    • hash (corresponding to the sought filexiv), followed by
    • uint8 (manifest packets sought count, 0=all), followed by
    • n* uint16 (manifest packet index sought), followed by
    • padding to Serpent-message length.
  • 4.4.b. File Transfer, manifest (always sent and only sent in response to ID 3)
    • uint8 (type ID, =4), followed by
    • uint16 (count of manifest packets for this filexv), followed by
    • uint16 (index of current packet in list above), followed by
    • uint8 (fragment countxvi), followed by
    • n* uint64 (hash of the nth fragment of manifested file).
    • uint16 (keccak hash of foregoing), followed by
    • padding to Serpent-message length.
  • 4.4.c. File Request, chunks
    • uint8 (type ID, =5), followed by
    • hash (corresponding to the sought file), followed by
    • uint8 (file chunks sought count), followed by
    • n* uint64 (the hash of fragment sought), followed by
    • padding to Serpent-message length.
  • 4.4.d. File Transfer, non-last chunk (always sent and only sent in response to ID 5)
    • uint8 (type ID, =6), followed by
    • chunk.
  • 4.4.f. File Transfer, last chunk (sent at most once per ID 3)
    • uint8 (type ID, =7), followed by
    • uint16 (bytesize of useful part of the chunk followingxvii, followed by
    • chunk.xviii
  • 4.5. Client Actionxix :
    • uint8 (type ID, =8), followed by
    • text (fully specified action, see section 7), followed by
    • uint16 (message count), followed by
    • padding to Serpent-message length.
  • 4.6. World Bulletinxx :
    • uint8 (type ID, =9), followed by
    • uint32 (id of top level itemxxi), followed by
    • uint8 (count of objects), followed by
    • object listxxii, followed by
    • uint16 (message count), followed by
    • padding to Serpent-message length.
  • 4.7. Object Request :
    • uint8 (type ID, =10), followed by
    • uint8 (count of objects), followed by
    • n*int32 (id of object), followed by
    • uint16 (message count), followed by
    • padding to Serpent-message length.
  • 4.8. Object Info :
    • uint8 (type ID, =11), followed by
    • uint8 (count of objects), followed by
    • n times uint32 (id of object) and text (object properties, as per extant game structures, including art files needed and so onxxiii), followed by
    • uint16 (message count), followed by
    • padding to Serpent-message length.
  • 5. RSA Packetsxxiv :

    • 5.1. RSA key set.xxv

      • uint8 (equal to 251 to indicate packet contains a new RSA key), followed by
      • uint8 (protocol version), followed by
      • uint16 (subversion), followed by
      • uint32 (IP of serverxxvi), followed by
      • uint32 (IP of clientxxvii), followed by
      • uint64 (keccak hash of client binary), followed by
      • uint64 (e of RSA key), followed by
      • uint8*490 (N of RSA key), followed by
      • uint64 (preferred padding -- the magic value of 0x13370000 requests random padding ; all other values will be used as such, bitwise, ie like an infinite-length OTP consisting of the value repeated), followed by
      • uint16 (message count), followed by
      • padding to RSA-message length, 1424 (5616-8-8-16-32-64-64-3920-64-16) bits exactly.
    • 5.2. Serpent key setxxviii :
      • uint8 (equal to 157 to indicate packet contains new Serpent keys), followed by
      • uint8 (count of keysxxix in this set, n ; n<=19xxx), followed by
      • n*(4*int64 + uint32) (32 bytes each key followed by a 4 byte ID calculated as crc32 on the key itself), followed by
      • an uint8 flag (LSB bit set -- keys to be used to talk to client ; MSB set -- key to be used to talk to server ; client-set MSB is ignored by server ; server will set LSB on keys requested by client for its own use thus supporting clients with no trustworthy random generators of their own), followed by
      • uint16 (message count), followed by
      • padding to RSA-message length.

    6. Protocol Mechanics :

    • 6.0. All communications between server and client will consist of messages. These messages may be encrypted either via eucrypt.RSA or eucrypt.Serpent. All RSA-encrypted messages will be exactly 1`470 bytes in length ; all Serpent messages will be exactly 1`472 bytes in lengthxxxi. The server will handle Serpent messages in preference of RSA messages (which are processed on an as-available basis). Clients that send garbage will be punished ; the costs involved (encryption/decryption ; generating entropy ; lookups and whatnots) will be pushed onto the client, for which reason writing the clients lightly pays off.
    • 6.1 The handshake works as follows :
      1. New client issues 5.1 packet keyed to the server's public key, including its own RSA key.
      2. The client's IP is recorded, and will have to be explicitly changed by the client later if needed. Server replies with 5.1 packet keyed to the client's announced key, including its private RSA key for use by that client ; and with 5.2 packet containing key material for client's use in keying messages to the server. If the client fails to provide its own set of serpent keys, the server will further issue it a set of serpent keys ; thenceforth the server will send more serpent keys mirroring the client's supply, and will similarily mirror key burning and select operations on its own set.
      3. Should the client's IP change, it will issue a 5.1 packet keyed to the server public key immediately followed by a 5.1 packet keyed to the server's original private key. The server will then update the client's IP accordingly (this also trashes the extant Serpent keyset and triggers 5.2).
      4. The bulk of communication is intended to go through the Serpent system ; outside of identification and bootstrap handshakes RSA isn't used. Should either party believe the Serpent keysets've been FUBAR'd, a 5.2 packet will reset that keyset.
    • 6.2. The server will issue type 4.6 packets in response to relevant type 4.5 packets received -- these can either signify the acceptance or the rejection of the client action, and the client must adjust its internal state accordingly.

    7. Character Actions :

    • 7.0. Lock :
      • uint8 (type ID, =0), followed by
      • uint8 (count of objects), followed by
      • n* uint32 (object ids). Defaults to currently targeted item.
    • 7.1. Make :
      • uint8 (type ID, =1), followed by
      • uint32 (object id, defaults to current target), followed by
      • uint32 (object id, defaults to current recipe in mind), followed by
      • uint32 (object id, defaults to currently equipped tool), followed by
      • uint8 (count of objects). The order for producing count items is entered into the queue (note that crafting only progresses if the player and/or hireling NPCs find themselves in certain situations re equipment, position, etc).
    • 7.2. Explore : uint8 (type ID, =2). The character will attempt to find some resources.
    • 7.3. Exchange :
      • uint8 (type ID, =3), followed by
      • uint32 (object id, the other party), followed by
      • uint32 (object id, the trade itselfxxxii), followed by
      • uint8 (count of objects), followed by
      • n* uint32 (object ids) and uint64 (object count), followed by
      • uint8 (flag, set to 0x10 to lock a trade and to 0x0c to approve a trade previously locked by both players).
    • 7.4. Attack :
      • uint8 (type ID, =4), followed by
      • uint32 (object id, the other party), followed by
      • uint32 (object id, the battle itself (server set, exactly in the way trade works). This not currently implemented, except player setting itself the bomb results in instadeath.
    • 7.5. Repair :
      • uint8 (type ID, =5), followed by
      • uint32 (object id, defaults to current target), followed by
      • uint32 (object id, defaults to current equipped tool).
    • 7.6. Move :
      • uint8 (type ID, =6), followed by
      • uint32 (destination id, defaults to current target), followed by
      • uint32 (slot id), followed by
      • uint32 (object id, of the item being moved), followed by
      • uint32 (quantity moved).
    • 7.7. Train :
      • uint8 (type ID, =7), followed by
      • uint32 (object id, the other party), followed by
      • uint32 (object id, the train session itself (server set, exactly in the way trade and battle work). This not currently muchly implemented, except some NPCs train for money -- but will get greatly expanded asap.
    • 7.8. Relocate :
      • uint8 (type ID, =8), followed by
      • object type, containing new client position.

      Please leave your comments below.

      ———
      1. Floating point item deliberately not specified. []
      2. Each such message is OAEP-padded and then encrypted with a (3920 bit) RSA key. Three such messages are strung together to form a RSA packet. Because of the significant overhead involved (both in terms of space and time), Serpent-encrypted comms are preferred whenever feasible. []
      3. See TMSR-RSA OAEP padding for the principle and this discussion for details. []
      4. This is the total size of a packet containing RSA-encrypted material. The useful size (ie payload) of such a packet is merely 702 bytes. []
      5. We really really want to keep this down. 13 bytes is the lowest I can conceive of, but I would so not mind halving it. []
      6. Representing the identifying hash of the object in question.

        We're using the narrower size to save on network traffic -- all the expenditure of another 32 bits here would buy us is de-ambiguation for cases where the count of objects around makes 1 in 2 billion collisions relevant. It doesn't seem likely a client could support such abundance of objects.

        Note that the hashes used here are client-specific, the server doesn't leak its own internal representation of objects to the clients. []

      7. Coordinates X, Y and Z in that order. Because the map goes from -500 to +500, the relationship between the given figure (GF) and map coordinates (MC) is GF / 65.535 - 500 = MC. []
      8. As a full rotation is 2 pi, the relationship between the given figure (GF) and object rotation (OR) is GF / 128 * pi = OR. []
      9. This arrangement permits the representation of arbitrarily large textfields (2nd segment can represent up to 115`792`089`237`316`195`423`570`985`008`687`907`853`269`984`665`640`564`039`457`584`007`913`129`639`936 bytes, which is more than enough space for all the text ever produced -- or likely to ever be produced -- by humanity) at the modest cost of a fixed 3 byte header.

        Unfortunately, it has no longer any utility for Eulora, since we've moved to fixed packets. I'm preserving it here because I really like it in the abstract and it has no other place to go. []

      10. These packets consist of 92 successive 128 bit chunks, Serpent-enciphered individually. To extract the payload one splits the message into 92 16-byte chunks, deciphers them then collates the output into a final result. To produce the packet one cuts a 11`776 bit payload into 92 128-bit chunks, Serpent-enciphers them, and collates the results into the outbound packet. []
      11. Polynomial generator 0x04c11db7. Keys with null IDs are discarded and regenerated. []
      12. Each client and the server will keep a count of messages they sent each other. This value must be incremented on each subsequent message sent by no less than 1 and no more than 255. []
      13. Keys are maintained by both client and server in an ordered ring buffer 256 elements long. The server will not send more keys than the total count of 0(absent)-keys in the respective buffer, irrespective of request count. If the message contains an unknown ID or otherwise is unprocessable, the issuance of a 5.2 packet is adequate response. []
      14. This is the keccak hash of the actual file contents. By convention this hash rendered as a 32 alphanumeric character string is also used as the filename for the file in question. []
      15. This system allows up to 65`536 manifest packets, adding up to potentially 11`993`088 (65`536 * 183) fragments representing a file of up to about 140 Gb (141`038`726`624 = 11`993`088 * 11`760 + 11`744 bits exactly). This will have to be sufficient. []
      16. From 1 to 146 inclusive. []
      17. This also means the protocol does not allow the transfer of files of certain sizes (within 8 bits of a multiple of 11760), which is fine with me. []
      18. The final fragment of the file will have to be padded to length as per this spec. []
      19. This is never issued by the server. []
      20. This is never issued by the client. []
      21. As discussed in comments, the world is a hierarchical structure of objects within objects. []
      22. This portion will get more clarification later on! []
      23. The complete list of these is currently exposed by the extant client, but in any case we'll publish a complete schematic. The server will set the "target" of the player on the last object in the list. []
      24. These packets consist of three 490 byte successive chunks RSA-encrypted individually. To extract the payload one splits the message into three 490 byte chunks, RSA-decrypts and de-OAEP-pads each one, the collates the results into a final result. To produce the packet one cuts a 5`616 bit payload into three 1`872 bit chunks, OAEP-pads and encrypts them, and collates the results into the outbound packet. []
      25. This is the manner in which new clients register their RSA key with the server (thereby opening a new game account). Later replacement of a registered key IS NOT POSSIBLE. Keep your client's RSA key safe.

        This is also the manner through which IP changes for an account are registered with the server. See the Protocol Mechanics heading for details. []

      26. This is used by the server when signalling to the client to talk to a different server (which is a thing for scaling, because different sectors will be handled by different servers). []
      27. If the client doesn't know its own IP, it's acceptable for this to be zero. []
      28. This permits either client or server to declare Serpent keys via RSA. It is not mandatory (as there exists a Serpent-encapsulated mechanism for the same end) but entirely legal. The server will always respond with at least one 5.2 packet after an accepted 5.1 packet creates a new player account, consisting of 40 Serpent keys to be used to talk to the server. Should the client respond with any other packet than 5.2 or 4.1, the server will send a 2nd 5.2 packet, containing 40 Serpent keys for the client's use. []
      29. Keys obtained through a 5.2 packet are always indexed in the client's buffer in the order they were found in that packet, starting with the first position. []
      30. A RSA packet has 702 total bytes available, of which 5 are used otherwise and the remainder of 697 are available for packing serpent keys, which take 36 bytes each (crc32 id inclusive). []
      31. Lenght being actually how they're sorted on the server side. []
      32. This is set by server through a type 6 message for both players involved, the trade is an object like any other that the OP has to request. The server will also expire trades, enforce them etc. []
    Category : S.MG  | 135 responses.