Chapter 3: Tacos!

My poker game was more or less done by now, so I had to start thinking about setting things up for an actual experiment! The experiment needs to be run in-lab, as opposed to online, because the participants will need to use their webcams correctly; random online people might point their webcam at the wall or at other undesirable places. To motivate people to come, we would need to pay them, so the ICT will need to grant us some funds to do so. To be allowed to run experiments on human subjects, I needed to submit a summary of my experiment for a thorough review to a certain organization, and finally, I would need to extensively test my program with enough people, which would serve as a general test for the experiment, but also as a stress test for the server, since it will be sending and receiving a lot of camera images.

Luckily, there is a bunch of very helpful people at ICT who were willing to guide me in all these areas. The Facilities department was able to get a classroom with 12 computers for me, called the TACL-lab. I have no idea what the abbreviation stands for exactly, but for easier reference, people call it the 'taco'-lab. Facilities was also able to provide a bunch of webcams and give me access to the taco-lab, so that was covered in a few days.

Taco-lab in action

Applying for money to pay study participants is not a big deal either, although it can take up to four weeks to come through. Writing a summary for review was a lot of work, but most of this work was done by a more experienced ICT member, I only needed to supply the basic information, so I am very grateful to her! Finally, with access to the taco-lab, I could grab some other interns and start testing my poker game!

Even though we initially only tested with 6 out of 12 available computers, we managed to crash the server after a couple of minutes of playing. The Tomcat function that I wrote to send video frames to the other player was synchronized, but only for each session. Synchronization means that only a single 'thing' is able to use this function at any time, any other use of this function by 'thing' will need to wait until the original user is done with it, this makes sure that only one 'thing' uses the WebSocket channel. However, the 'thing' in my case was a session, which essentially means a single computer. In a scenario where only 2 people are playing each other, I wouldn't have noticed this, because the images are sent very quickly. During the stress test, 6 sessions are using this method simultaneously, so at some point two different sessions would access this function at exactly the same time, (virtually) exploding their connections.

Some people had some significant feedback about the workings of the game as well, such as the fact that they didn't really have time to look at their opponent's facial expressions, on which they are partly supposed to make their betting decisions, because of the strict timer that only gives you 7 seconds to place your bet. They were also not quite sure how the game worked before they played a couple of rounds, so they suggested making a tutorial system, and one of the participants never played any kind of poker so she didn't understand the basic explanation that I had given before starting the game. All around, the presentation of information was lacking, so I set out to fix just that...

Chapter 2: Work work...

One of the people here at ICT was already working on a similar project where he needed to record audio and/or video while people were playing a simple game. He provided me with the framework that he had built so far and gave me some valuable tips in what to look out for. This framework was built in Java using a Tomcat server, with access to a MySQL database and it used a WebSocket connection to send game events and the SimpleWebRTC framework to send video and audio to the other player. In order to analyze the video and/or audio, these fragments were also manually sent to the Tomcat server for storage.

Using the information that this person had given me, I started building the back-end for my poker game in Java for use on the Tomcat server. I submitted a request to our IT department to set up a Tomcat server for me, which they supplied a couple days later. It took me about two weeks to get a working prototype, where two people could connect to the server and the server would connect these players together in a game, where their webcam images would be sent using the SimpleWebRTC framework. This prototype did not store any information yet, however.

Example of poker game in action

I sincerely disliked the fact that the webcam video was essentially going to be sent twice; once to our Tomcat server for storage, and once to the other player. I wanted to find a way to only send the webcam images to the Tomcat server. This server would then store the image and send a copy to the other player. The SimpleWebRTC framework only allows you to send the images to another WebRTC-enabled client, which is usually only available in JavaScript. There is a WebRTC framework available in Java (called IceLink), but this seemed quite complicated to set up properly. Eventually I decided to use a more basic approach. I will capture just about every webcam video frame and send this frame in base64 encoding to our Tomcat server (using POST). The Tomcat server will store this image as a JPEG file and send the encoded image to the other player using the already existing WebSocket connection. The receiving client would simply display the incoming image on screen.

Surprise!

This solution seemed to work fine for a while, but after about 10-15 seconds the WebSocket connection would crash, which meant that both the video and the game events would not work anymore. After a more thorough investigation I found that the connection would only crash whenever the client tried to send a game event to the server, such as placing a bet. I've never conclusively found what the reason for the crash was, but I expect that the crash happened because the WebSocket connection was busy receiving an image file, while the game is trying to send a bet placement message, which cannot be done concurrently, so it simply throws an error and stops altogether. By using a seperate WebSocket connection that is only to be used for video, this problem seemed to be resolved and the connection does not crash anymore.

Chapter 1: Pokerface

At the start of my internship I needed to get used to the environment and meet all kinds of different people who I might be working with. It quickly became clear that I needed to figure out what kind of project I wanted to do here; they gave me freedom to do whatever I wanted, as long as it fit in with recent ICT studies.

As stated in the introduction, one of their studies looked at the influence of a virtual human’s facial expressions on a human player while playing a game called Iterated Prisoner’s Dilemma. However, the avatar only showed a brief facial expression after a round was done; it was not possible to infer the computer’s decision based on his face. If two human players were playing this game, one of the players might be able to guess what decision the other player has made based on an (involuntary) expression.

Source: C. De Melo, L. Zheng and J. Gratch, "Expression of Moral Emotions in Cooperating Agents," in 9th International Conference on Intelligent Virtual Agents, Amsterdam, 2009. 

I decided to focus more on that part of facial expressions in games. However, the Iterated Prisoner’s Dilemma is not an ideal scenario to look for facial expressions, because a player’s goal is not very clear. If a player shows a genuine smile, the meaning of this smile is still ambiguous; he/she might smile out of either competition or cooperation. In order to remove this ambiguity I needed to design a new game where a player’s goal is always obvious. Since we will be looking at facial expressions during gameplay, I started thinking of bluffing in a poker game, so why not let people play a game of poker against each other?

Unfortunately there are a couple of good answers to this simple question. We would need to take a player’s existing poker playing experience into account, especially when an experienced player is matched against an inexperienced player. There are a lot of different variations of poker, and even though Texas Hold’Em is considered to be the most popular by far, it is also the most complex to analyze; we want to be able to tell what a person is thinking when he/she makes a decision.

To remedy or reduce all of these problems I designed a new game that is a drastically simplified version of poker. Players receive a single card between 2 and 10 and they only have 4 different betting options: No bet, low bet, medium bet or high bet. Additionally, the players will have a limited time to make their betting decision, but this betting decision will only be revealed to the other player when this time has passed. This is done to reduce an experienced player’s ability to ‘intimidate’ a less experienced player by placing bets instantly. The players will play for a limited amount of rounds and will start out with enough chips to place high bets every round if needed. We will record all facial expressions during the game and analyze these to possibly find differences between bluffs, tells and general smiles. This information can then be used for a more realistic virtual human playing a similar pure-competitive game. A quick and dirty preview:

This preview has a working server back-end (meteorJS) that sends and receives messages from the Javascript front-end. The card is made completely in CSS with fancy transforms that allow you to rotate the card to put it face down.