Client / Server Development

This page details the current (experimental, partial) implementation, and plans for further development.

Feel free to comment, criticize, add ideas, and so on.

Some code can be found at http://repo.or.cz/w/crawl/crawl-nettiles.git. The map_knowledge/map_cell refactoring, erased items and complete monster_info have been merged into crawl, though the tiles rendering does not yet fully make use of this.

First milestone: NetTiles console parser

An already existing solution is the NetTiles branch of crawl, which offers the ability to connect to console servers, and display the game in a tiles-like interface, using heuristic parsing methods.

While this is usable, the console interface is limited, and does not allow to provide a really good Tiles experience.

For instance, item tiles cannot be precisely assigned, it is not possible to tell whether ground or water (or stairs, etc.) is under something, altars cannot be immediately correctly determined, monster tiles don't always work well (e.g. no health is available if multiple monsters of the same type are visible), and so on.

Second milestone: first client/server solution

Refactoring

The first idea is to refactor the codebase so that structures reflecting player information are readily available and are used internally to produce both the console and tiles view.

This has also the advantage of preventing accidental leaks and simplifying the code.

Items

For items, a function is provided to create a new item_def (typedef'ed to item_info) from the current item_def structure containing only player known information.

Items have the unique property that they can be fully known if identified, which makes this approach appropriate.

Ideally, there should be empty derived class of a base item info structure to enforce type checking, but this has currently not been done.

Monsters

For monsters, the monster_info structure used by the monster list has been expanded to cover all visible information about a monster.

A completely different structure is appropriate here, because monsters have both detailed properties (e.g. HP, AC, EV) and artificial intelligence data, none of which should be exported to the player as-is.

monster_info additionally holds smart pointers to item_info structures for the visible equipment.

Map cells

As a preliminary step, env.show has been removed in favor of storing both the visible and remembered squares in env.map_knowledge: this will simplify the code by having all map data in a single place.

It also allows to easily experiment with infinite LOS and other different LOS models.

The map_cell structure, currently used for remembered cells, has been/is going to be expanded to contain all information on a given square.

env.tile*_[bf]g has been removed in favor of generating the tiles view from env.map_knowledge.

If this has indeed been done, the code does not appear to be available. — rob 2010-09-07 17:58

map_cells optionally hold smart pointers to item and moster information if those are present on the cell

Player

This has not been done yet. The idea is to initially use the existing serialization function, which however leaks information.

The best plan seems to be to create either a function to “erase” data from the player structure, or directly create another set of serialization functions.

Actually using a separate structure seems likely a bad idea at least initially, and it seems better to instead put “fake” data in struct player as necessary (e.g. fixed fake durations for active effects).

Coordinates

Currently coordinates are not exposed. To expose them without any information leak, the idea is to use a coordinate system where (0, 0) is set to the first square the players enters the level on.

Monster ids

To expose monster ids, they should be generated on first exposure to the user, so that gaps don't indicate generation of unseen monsters.

Server architecture

For the first approach, the best option seems to have an executable which both a “normal” crawl console version (but with the refactoring above applied), but also listens on a unix socket or tcp port, allowing multiple clients to connect and get game updates.

Input is going to be handled with stdin as usual at first.

This works by adding an option called ”–server” which causes the crawl executable to listen on an unix socket (–server local:/tmp/crawl-server), a tcp port (tcp:1234) or IPv6 tcp port (tcp6:1234).

Incremental update protocol

The idea of the Crawl protocol is that every client connected starts by getting a “full update” containing the full user-visible Crawl state at the moment the client connects.

After this initial full updates, all client are sent the same “update stream”, which consists of a stream of packets which incrementally update the client state to reflect changes in the game world

Server implementation

The current implementation is based on Boost Asio and uses 3 different threads in the process and a global fair mutex, the “Big Crawl Lock”.

The update_stream_distributor is the thread-safe component responsible for distributing full updates and incremental updates to the connected clients, which are added to it by the update servers components, which asynchronously listen to the server sockets. It is invoked in all 3 threads.

The first thread is the game thread, which runs the game code as it is currently done, which the crucial changes that it holds the “Big Crawl Lock”, and releases it around delays and waits for input and that every time viewwindow() is called, it constructs an incremental update and gives it to the update stream distributor.

The second thread is the Boost Asio I/O service thread, which runs asynchronous I/O callbacks (boost::asio will use epoll on Linux for this). In this thread new connections are accepted, and the initial full update and later the update stream are sent to writable sockets.

The third thread is the full update thread: this thread sleep until there is a new client that needs a full update, the grabs the Big Crawl Lock, creates the full update data and gives it to the update stream distributor.

Client architecture

The client should reuse the NetTiles interface and code.

Debug version

The debug version will take a command line option with the same syntax of ”–server” and will connect there, as well as a –sendinput option to specify a place to send input to.

End-user version

The PuTTY-based terminal should be replaced with a simple ASCII-only terminal emulator used to parse and display dgamelaunch.

It will connect via SSH to a supported server, and set a special environment variable using the SSH protocol.

If the servers supports it, it will send a special magic signature as well as the path of the unix socket or TCP port to connect to on the server.

It will then create a separate SSH channel in the SSH connection, and ask it to be connected to that port or TCP socket, and will receive the protocol update stream there.

Dgamelaunch architecture

dgamelaunch or possibly supporting configuration will need to be modified to deliver the update stream to the clients instead of the TTY data.

In particular, it should create all games specifying the –server option to have crawl listen an unix socket.

If the special environment variable is specified, instead of sending the game stdout to the player/watcher, it should send the path of the unix socket.

A special shell should also be allowed, which allows to read from one of these sockets, specified as a parameter.

Alternatively, a TCP port can be reserved, and the SSH port forwarding mechanism used to connect to it.

Third milestone: object-oriented Crawl

After a first solution is working, the Crawl codebase should be refactored, so that the game, the clients and the servers are all easily instantiable C++ objects, the source code of which is in separate subdirectories.

Executables can then be created by combining components to either make a client+game local all-in-one process, a console client+network server+game process, and so on.

Fourth milestone: fully semantic protocol with non-C++ bindings

Later, the protocol should be changed to only deliver semantic information about the game, making all examination commands, display and targeting logic run separately.

The protocol shuld also be changed to use a standard multilanguage serialization infrastructure instead of tags.cc (e.g. Google protocol buffers), so that bindings for other languages than C++ can be readily generated.

Fifth milestone: new clients

Finally, new clients can be added.

An HTML/JavaScript based client is the obvious addition. Options could be:

  1. Create an additional server component talking an HTML/JS specific protocol, plus the server website offering the HTML client.
  2. Automatically translate (probably inefficiently) the C++ code to JavaScript (e.g. C++→LLVM→JS backend) and thus run the HTML/JS client on the binary semantic Crawl protocol.
  3. Rewrite the client-related C++ code (e.g. tile packing) in Java, and either automatically translate to JavaScript (should work well), or natively compile with GCJ for “standard” crawl executables
  4. Write ad-hoc JavaScript using generated JavaScript protocol bindings to create the HTML/JS interface

Crawl bots should also be easily writable in several programming languages given all the previous work, and might be useful for automated testing.

Online Tiles

General Approach

Henry's proposal:

  1. Define a very strict format to describe game state (JSON Blob below).
  2. When the game state is changed (near the end of world_reacts() possibly?) query all the different state objects to fill out that JSON blob and send it over
  3. Build an extremely light parser for that which can pass the data back to the gui client or a website.

Details:

  • At end of main.cc::world_reacts(), converted player knowledge to JSON object for consumption by an online client. Player knowledge includes 'env.map_knowledge', and …?
  • Client will need to map 'map_cell' to a tile index with a function like this: tilepick.cc::tileidx_from_map_cell(tileidx_t *fg, tileidx_t *bg, const map_cell &cell)
  • Client also needs these enums to properly implement tileidx_from_map_cell:
    • enum.h:: enum dungeon_feature_type (this is “ft” (aka “feat”) in “cell” of “map_knowledge”)
    • enum.h:: enum monster_type (this is monster base type)
  • Some piece of showsymb.cc:: get_cell_show_class() will be required in the client.
  • Every displayed tile can have a foreground and backgroud tile to display (floor and item/monster/cloud).
  • QUESTION: tilepick.cc is seriously complicated. It would be easier to pass Tile indexes around, but the client would know much less and be tied to the Tile version's tile. Is passing feature indices around worth the trouble? I think so, but re-writing tilepick.cc in Javascript will be a pain.

> Passing the feature is vital, it's needed for bots and alternate UIs, which are a good part of client/server split. We could pass both that and the tile stack, though. — KiloByte 2011-04-14 10:28

  • Need TILE_GOLD type C++ enums output as Javascript enums. This can be done by messing with the Tile generation utility.
  • Need MI_STONE type C++ enums output as Javascript enums. This needs something clever, like a C++ enum parser or using MACROs to define the enums in enum.h.

JSON Blob

{"state" : "playing"},  // (e.g.initial splash screen, character selection, playing, dead, etc).  'splash_screen'
{"map_knowledge":       // **from env.map_knowledge**
  [     // Array of map_cell's with relevant data.

    // Only include map_cell if flags is non-zero.
    // Include XY so we don't have to have them all.
    {"cell":{"x":65,"y":8,
      "fl":2,          // flags
      "ft":67,         // feature
      "ft_colour":0,  // Only include if non-zero.
      "cloud":0,"cloud_colour":0}},  // Include cloud if non-zero.

      {"item":                       // item if present.
          "bt":4,                       // base_type
          "st":0,                       // sub_type
          "sp":0,                       // special
          "cl":0                        // colour
          "fl" : 0                      // flags
      },

      {"monster":  186                  // monster if present
/*
          "???base_type":4,
          "???sub_type":0,
          "???special":0,
          "???colour":0
*/          
      }

  ]
},

"texts": { 
         "MsgBox": [
                       {"line" : "first (or last line) from message box", "ch" : 7, "pa" : 0 },       // Bottom message box, get_last_messages(ncount)
                       {"line" : "next line from message box", "ch" : 7, "pa" : 0 },                  // "ch" is the channel. "pa" is the param.
                   ]                       
         // Are there other types of texts to include?
    },

"you" :                 // from you variable.
  {
    "name" : "Fool",
    
    "hp" : 1,           // HP Variables.
    "hpmax" : 1,
    
    "mp" : 1,          // MP variables
    "mpmax" : 1,
    
    "x" : 50,         // you.position
    "y" : 10,
    
    "title" : "the skirmisher",
    "species" : "Kobold",
    "job" : "Berserker",
    "god" : "Trog",
    "piety_rank" : 0,    // E.g. '*......'   piety_rank()
    
    "ac" : 1,               // you.armour_class()
    "ev" : 1,               // player_evasion()
    "sh" : 1,               // player_shield_class()
    "xl" : 1,               // you.experience_level
    "xp" : 0,               // you.exp_available
    "gp" : 0,               // $  you.gold

    "turns" : 1,            // you.num_turns
   
    // stats
    "str_max" : 10,     // you.max_stat(STAT_STR)
    "str" :      9,     // you.stat(STAT_STR, false)
    "int_max" : 10,
    "int" :      9,
    "dex_max" : 10,
    "dex" :      9,
    
    "wp": "knife",         // you.weapon->name(DESC_INVENTORY)
    "qv": "stones",         // you.weapon->name(DESC_INVENTORY)

...
  },
  "inventory" :        // from you.inv???
  [
    {"item" : ...},
    {"item" : ...}
  ]
  }


...
};

Example:

{"map_knowledge": [{"cell":{"x":7,"y":7,"fl":2,"ft":7,"feat_color":0}},
{"cell":{"x":57,"y":12,"fl":2,"ft":7,"feat_color":0}},
{"cell":{"x":7,"y":13,"fl":2,"ft":7,"feat_color":0}},
{"cell":{"x":8,"y":13,"fl":2,"ft":67,"feat_color":0,"item":{"bt":6,"st":0,"sp":116,"cl":7}}},
{"cell":{"x":9,"y":13,"fl":2,"ft":67,"feat_color":0}},
{"cell":{"x":17,"y":46,"fl":2,"ft":67,"feat_color":0}},
{"cell":{"x":18,"y":46,"fl":2,"ft":67,"feat_color":0,"monster":13}},
{"cell":{"x":19,"y":46,"fl":2,"ft":67,"feat_color":0}},
{"cell":{"x":20,"y":46,"fl":2,"ft":67,"feat_color":0}},
{"cell":{"x":21,"y":46,"fl":2,"ft":67,"feat_color":0}},
{"cell":{"x":22,"y":46,"fl":2,"ft":67,"feat_color":0}},
{"cell":{"x":23,"y":46,"fl":2,"ft":67,"feat_color":0}},
{"cell":{"x":24,"y":46,"fl":2,"ft":67,"feat_color":0}},
{"cell":{"x":45,"y":62,"fl":66,"ft":7,"feat_color":0}}
],
"texts" : { "MsgBox": "Trog says: Kill them all!<br>Press ? for a list of commands and other information.<br>A goblin comes into view. It is wielding an orcish dagger.<br>Save game and exit? <br>Welcome back, Fool the Minotaur Berserker.<br>Trog says: Kill them all!<br>Press ? for a list of commands and other information.<br><br>"},
"you":{"name":"Fool","hp":8,"hpmax":17,"hpbase":5000,"hpbase2":5017,"mp":0,"mpmax":0,"mpbase":5000,"mpbase2":5000,"ac":2,"ev":12,"sh":0,"xl":1,"xp":0,"gp":20,"turns":482,"x":40,"y":56,"title":"Chopper","species":"Minotaur","god":10,"job":"Berserker","piety_rank":2,"wp":"a - a +0,+0 hand axe","qv":"c - 2 darts"}
}

What's missing:

  • Player information
  • Monster information (will be handled like item as a part of the map_knowledge).
  • Inventory
  • Messages: use get_last_messages(ncount)

Status:

  • 4/11/11: Have C code in Crawl Tiles that will output the example JSON above at each world_reacts() call.
  • 4/12/11: Have Javascript code that will parse and display (improperly) the above example JSON code in a web page.
  • 4/15/11: Proof-of-concept code running here: http://dcssjsc.99k.org/Page4-Timer.htm Runs on 20 saved frames, no interactivity, dungeon and monsters rendered correctly, player not rendered, items not correct.
    • I get a large but very blurry critter with a whip and a cockroach in the left right corner staying and sometimes switching places, and another pair on the map. The map and tiles on it are reduced to a small size but otherwise render correctly except the very last frame. Items seem to work. — KiloByte 2011-04-16 03:33
  • 4/19/11: Updated proof-of-concept with minor changes: Displayed only part of the map. Player info is formatted, but not read yet. User must press a button to advance to the next frame. Monsters are rendered 50% correct.
Logged in as: Anonymous (VIEWER)
dcss/brainstorm/internal/clientserver.txt · Last modified: 2011-12-20 20:03 by XuaXua
 
Recent changes RSS feed Donate Powered by PHP Valid XHTML 1.0 Valid CSS Driven by DokuWiki