Chapter 6: Synchronizing and combining

Timestamping every frame and game event seemed like a good idea at first. Unfortunately, I had not realized that these timestamps are created based on the local time of the machine running the experiment. I had expected this local time to be more-or-less equivalent between the 5 available machines that were used for the experiment, but this did not seem to be the case. Some machines were up to 4 seconds off! Most of the experiments have already been run, and it would be a shame to let all this data go to waste, so I needed to find a way to fix time.

When I was setting up the database to store all game events, I added a field to this database table that would automatically set itself to the current time value whenever a new event occurred. This time is actually consistent, since this is always the local time of the database. Perfect, let's use that!
No, it's not supposed to be this easy: This time stamp was rounded to the nearest second, so according to the database, two people could perform an action at 'exactly' the same time. I wanted the timings in my program to be more precise than that. After some brainstorming I figured I could look at the 'imprecise' time of the database entry and compare that with the 'exact but not synchronized' time of the local time of the machine creating the event. Averaging over all events for a particular player would give me an estimate of the offset between the database time and the player time. I could determine this offset for both players and subtract these offsets from the original player time to get an estimated exact synchronized database time.

This is what I ended up doing, which took me quite some time to implement just right, but the results were definitely worth it. In some videos it was possible to see me walking around in the experiment room on both videos simultaneously, which was the perfect test to see if these offsets were calculated correctly. Luckily, they were, and I could see myself in the same awkward pose in both videos at the same time, yay me!

In the meantime, my supervisor noticed the tool I was building and had a bunch of ideas of different kinds of information he would like to see on it, such as a 'play' button instead of only the slider at the bottom, an option to hide unimportant game events, the results of the questionnaires for each participant, etc.

Now that most of my planned experiments were done, I could start running my videos through a number of facial expression extraction tools that ICT has available. These tools are able to track the position and rotation of the face, as well as detect different kinds of emotions that the participant might display. These tools created a huge amount of data for each video, and since we have over 200 videos in total, this is a lot of data to process. I started writing up some MatLab scripts to iterate over these huge files and average some meaningful data over different time spans during the game to produce a bunch of graphs; such as a graph for the smile levels during the game for each participant. These simple graphs don't provide too much information though, and it didn't take long before I was writing code to create a lot more complicated graphs...

Although these graphs can be useful to see possible trends, they're not exactly proof of anything. In order to prove that people smile more in the earlier rounds than in the later rounds, I'll need to run some fancy statistical analyses. I'm already looking forward to it...

Chapter 5: Balls of Steal

I took a break from the research right after running the batch of experiments that I talked about in the previous blog. With a couple of friends visiting from the Netherlands, we went on a road trip through California, Nevada and Arizona. You can read all about it in my upcoming (Dutch) blog post on my other blog: la-pruik.blogspot.com

When I came back from this trip I noticed a couple of emails in my inbox that talked about a British game show, called Golden Balls. At the end of this game, the two remaining participants 'fight' for a rather large sum of money, usually worth over $50,000 USD. Each of the participants receives a ball with the word STEAL written in it, and another ball with the word SPLIT written inside. The players can have a little discussion with each other to help them decide which ball to pick, and as soon as they have both picked their desired ball, their choice is revealed to the other (and to the audience).

The rules are as follows: If both players choose to SPLIT, the large sum of money is evenly divided among both players. If one of the players chooses to SPLIT, but the other chooses the STEAL ball, the player who chose STEAL will steal all the money, and the player who picked SPLIT will go home with nothing. However, if both players are greedy and choose to STEAL, both players will receive nothing.

My supervisor thought this would be a more interesting game than the previously implemented investment game. The possible choices are a lot clearer, and STEALing has a more negative connotation, so players are less likely to STEAL, even though it can give you the largest profit. The rules as stated above do not completely fit the definition of a Prisoner's Dilemma, however, so we changed it slightly so that when both participants choose to STEAL, they will still receive a very small reward - satisfying the Prisoner's Dilemma requirements.

After performing some JavaScript and CSS magic, I managed to simulate some balls that looked like they could open and close and the game looked like the example image shown below.

More experiments were set up, and I ran about 50 more people through this new experiment, which means I have a LOT more data to look through and analyse... I've been working on a custom tool to help me with that, since all of the game events are stored in a database, and the only way to display those events in a clear manner is with a custom-made application.

The goal of this video analyzing tool is to display the two participants playing with/against each other, with a slider below the videos to indicate the current time. Underneath that slider are some markers for the different events that occur during play. One of the earliest prototypes of this tool can be seen below, this did not yet have video for the opponent, but this expression occurred right after losing an attempted bluff.

I expected that synchronizing the video of both participants would be relatively easy, since every recorded frame and every game event is timestamped. However, this timestamp is created by using the client's local time, since this is the time the video frame is created and this makes sure that any lag from sending the frame to the server can be ignored, so simply comparing these timestamps should give me the precise time each frame and event occurred.

I was wrong. I was very wrong...

Chapter 4: Human experiments... for Science!

It's been over a month since my last post here already! It's been rather hectic the last couple of weeks, mostly because I've been working on the project, but also because most of the interns will be leaving ICT soon, and everyone is trying to cram in some fun events during the little free time that we have left.

Last month I talked about needing to improve the presentation of information in the poker game. With the help of some ICT people the interface had been improved a bunch and the explanation of the games presented to the player is now a lot clearer. My supervisor (Jon Gratch) wanted to recreate an experiment that one of his students had done a while ago, so he asked me to add another game to my experiment, called the Investment game. This investment game is basically the same as the Iterated Prisoner's Dilemma, where each player has two choices each round. The combination of the two players' choices determine the amount of points both players receive.

As you might be able to see, I used the same dashboard element from the poker game in this game, replacing the amount of chips by the score of the players. Most of the server-side code from the poker game was generic enough that it was rather easy to rewrite it to work for the Investment game.

The experiment was nearing completion now, but it needed one more thing before I could start inviting participants: a bunch of surveys. Before the participants would start any of the games, they will need to fill in a number of surveys to determine a number of personality traits. The surveys we were supposed to use were the following four:
- DANVA
- SVO
- Mach-IV
- ERQ

I won't bore you with the details of each of those surveys, if you wish to know more, Google will be able to tell you plenty. In short, these surveys tell if you are a competitive person and whether or not you are able to perceive and/or display emotions. After each game, the participant was also asked to fill in a little survey about his/her experience with the game. ICT uses a website called Qualtrics to easily create, edit and share these kinds of questionnaires and all of these required surveys were readily available for me to use. I just needed to set up the Survey Flow so participants could seamlessly go between my games and the Qualtrics questionnaires.

As happens often with programming jobs, Jon Gratch announced last-minute that he would like to implement two different conditions. Some of the participants would not be able to see each other while playing the games, while other participants would play as usual. The recording of facial expressions would be enabled at all times, but some of them would not see their opponent. When players believe they are not directly being watched, they might be more inclined to show facial expressions, even though they know they are being recorded. Easy fix: simply draw a blank rectangle over the area where the opponent's face is supposed to be in the games with the no-camera condition.

Participants were recruited from Craigslist and promised a $30 reward for an experiment lasting approximately 60 minutes. We created a total of 40 possible time slots where people could sign up for, these sign-ups were completely full within two hours of posting on Craigslist - apparently these experiments are popular. Approximately 30 out of 40 people actually showed up. Whenever someone didn't show up, I would need to use an ICT confederate to step in in order to maintain an even amount of participants.

Now that all of the experiments are finished (for now), I'll need to analyze all of the gathered data and try to make sense of it!

Chapter 3: Tacos!

My poker game was more or less done by now, so I had to start thinking about setting things up for an actual experiment! The experiment needs to be run in-lab, as opposed to online, because the participants will need to use their webcams correctly; random online people might point their webcam at the wall or at other undesirable places. To motivate people to come, we would need to pay them, so the ICT will need to grant us some funds to do so. To be allowed to run experiments on human subjects, I needed to submit a summary of my experiment for a thorough review to a certain organization, and finally, I would need to extensively test my program with enough people, which would serve as a general test for the experiment, but also as a stress test for the server, since it will be sending and receiving a lot of camera images.

Luckily, there is a bunch of very helpful people at ICT who were willing to guide me in all these areas. The Facilities department was able to get a classroom with 12 computers for me, called the TACL-lab. I have no idea what the abbreviation stands for exactly, but for easier reference, people call it the 'taco'-lab. Facilities was also able to provide a bunch of webcams and give me access to the taco-lab, so that was covered in a few days.

Taco-lab in action

Applying for money to pay study participants is not a big deal either, although it can take up to four weeks to come through. Writing a summary for review was a lot of work, but most of this work was done by a more experienced ICT member, I only needed to supply the basic information, so I am very grateful to her! Finally, with access to the taco-lab, I could grab some other interns and start testing my poker game!

Even though we initially only tested with 6 out of 12 available computers, we managed to crash the server after a couple of minutes of playing. The Tomcat function that I wrote to send video frames to the other player was synchronized, but only for each session. Synchronization means that only a single 'thing' is able to use this function at any time, any other use of this function by 'thing' will need to wait until the original user is done with it, this makes sure that only one 'thing' uses the WebSocket channel. However, the 'thing' in my case was a session, which essentially means a single computer. In a scenario where only 2 people are playing each other, I wouldn't have noticed this, because the images are sent very quickly. During the stress test, 6 sessions are using this method simultaneously, so at some point two different sessions would access this function at exactly the same time, (virtually) exploding their connections.

Some people had some significant feedback about the workings of the game as well, such as the fact that they didn't really have time to look at their opponent's facial expressions, on which they are partly supposed to make their betting decisions, because of the strict timer that only gives you 7 seconds to place your bet. They were also not quite sure how the game worked before they played a couple of rounds, so they suggested making a tutorial system, and one of the participants never played any kind of poker so she didn't understand the basic explanation that I had given before starting the game. All around, the presentation of information was lacking, so I set out to fix just that...

Chapter 2: Work work...

One of the people here at ICT was already working on a similar project where he needed to record audio and/or video while people were playing a simple game. He provided me with the framework that he had built so far and gave me some valuable tips in what to look out for. This framework was built in Java using a Tomcat server, with access to a MySQL database and it used a WebSocket connection to send game events and the SimpleWebRTC framework to send video and audio to the other player. In order to analyze the video and/or audio, these fragments were also manually sent to the Tomcat server for storage.

Using the information that this person had given me, I started building the back-end for my poker game in Java for use on the Tomcat server. I submitted a request to our IT department to set up a Tomcat server for me, which they supplied a couple days later. It took me about two weeks to get a working prototype, where two people could connect to the server and the server would connect these players together in a game, where their webcam images would be sent using the SimpleWebRTC framework. This prototype did not store any information yet, however.

Example of poker game in action

I sincerely disliked the fact that the webcam video was essentially going to be sent twice; once to our Tomcat server for storage, and once to the other player. I wanted to find a way to only send the webcam images to the Tomcat server. This server would then store the image and send a copy to the other player. The SimpleWebRTC framework only allows you to send the images to another WebRTC-enabled client, which is usually only available in JavaScript. There is a WebRTC framework available in Java (called IceLink), but this seemed quite complicated to set up properly. Eventually I decided to use a more basic approach. I will capture just about every webcam video frame and send this frame in base64 encoding to our Tomcat server (using POST). The Tomcat server will store this image as a JPEG file and send the encoded image to the other player using the already existing WebSocket connection. The receiving client would simply display the incoming image on screen.

Surprise!

This solution seemed to work fine for a while, but after about 10-15 seconds the WebSocket connection would crash, which meant that both the video and the game events would not work anymore. After a more thorough investigation I found that the connection would only crash whenever the client tried to send a game event to the server, such as placing a bet. I've never conclusively found what the reason for the crash was, but I expect that the crash happened because the WebSocket connection was busy receiving an image file, while the game is trying to send a bet placement message, which cannot be done concurrently, so it simply throws an error and stops altogether. By using a seperate WebSocket connection that is only to be used for video, this problem seemed to be resolved and the connection does not crash anymore.

Chapter 1: Pokerface

At the start of my internship I needed to get used to the environment and meet all kinds of different people who I might be working with. It quickly became clear that I needed to figure out what kind of project I wanted to do here; they gave me freedom to do whatever I wanted, as long as it fit in with recent ICT studies.

As stated in the introduction, one of their studies looked at the influence of a virtual human’s facial expressions on a human player while playing a game called Iterated Prisoner’s Dilemma. However, the avatar only showed a brief facial expression after a round was done; it was not possible to infer the computer’s decision based on his face. If two human players were playing this game, one of the players might be able to guess what decision the other player has made based on an (involuntary) expression.

Source: C. De Melo, L. Zheng and J. Gratch, "Expression of Moral Emotions in Cooperating Agents," in 9th International Conference on Intelligent Virtual Agents, Amsterdam, 2009. 

I decided to focus more on that part of facial expressions in games. However, the Iterated Prisoner’s Dilemma is not an ideal scenario to look for facial expressions, because a player’s goal is not very clear. If a player shows a genuine smile, the meaning of this smile is still ambiguous; he/she might smile out of either competition or cooperation. In order to remove this ambiguity I needed to design a new game where a player’s goal is always obvious. Since we will be looking at facial expressions during gameplay, I started thinking of bluffing in a poker game, so why not let people play a game of poker against each other?

Unfortunately there are a couple of good answers to this simple question. We would need to take a player’s existing poker playing experience into account, especially when an experienced player is matched against an inexperienced player. There are a lot of different variations of poker, and even though Texas Hold’Em is considered to be the most popular by far, it is also the most complex to analyze; we want to be able to tell what a person is thinking when he/she makes a decision.

To remedy or reduce all of these problems I designed a new game that is a drastically simplified version of poker. Players receive a single card between 2 and 10 and they only have 4 different betting options: No bet, low bet, medium bet or high bet. Additionally, the players will have a limited time to make their betting decision, but this betting decision will only be revealed to the other player when this time has passed. This is done to reduce an experienced player’s ability to ‘intimidate’ a less experienced player by placing bets instantly. The players will play for a limited amount of rounds and will start out with enough chips to place high bets every round if needed. We will record all facial expressions during the game and analyze these to possibly find differences between bluffs, tells and general smiles. This information can then be used for a more realistic virtual human playing a similar pure-competitive game. A quick and dirty preview:

This preview has a working server back-end (meteorJS) that sends and receives messages from the Javascript front-end. The card is made completely in CSS with fancy transforms that allow you to rotate the card to put it face down.

Introduction

Hello and welcome to the amazing blog by some guy called Jeroen! As a final project for my Master's degree I decided to go to the USA for a research internship, and as it turns out, a research institute of the University of Southern California in Los Angeles was willing to put up with me. This institute is called ICT - Institute for Creative Technologies.

Before my departure to the United States I had a few email conversations with my supervisor-to-be, professor Jonathan Gratch, with regards to the nature of the internship. He had already done quite some research into the impact of facial expressions of a virtual character on a human when someone was playing a game against this personal character. However, the game that was used for all of these experiments - Iterated Prisoner's Dilemma - can be played in different ways, the real goal of the game isn't very clear. This means that the motives of the virtual character could be interpreted in multiple ways when it is displaying some kind of facial expression.

My general project description would be to look closer into facial expressions while playing a game and for my personal interest I would like to focus more on the gaming aspect. The result of the email communication with prof. Gratch was the following broad project description:

In many social situations, another person’s expressed emotion affects our impression of the motives behind his or her actions.  We trust others partly on the basis of facial appearance, including its emotional aspects, partly on the basis of whether signaled emotions seem sincere, and partly on the basis of a sense of emotional rapport.  In these and other ways, emotional information can help to co-ordinate different people’s actions and intentions.  However, people can also manipulate their emotion presentations in order to conceal or disguise their motives. In this project, we will uses computer technology to gain insight into these emotional processes. 

One or more simple games (e.g., prisoner’s dilemma) will be created where the face and choices of both players will be recorded as they play. Computer analysis of these expressions will provide a grasp of which facial expressions result in a certain decision being made. In another set of experiments we will use these expressions to purposefully manipulate the other player into a certain decision.

 In this blog I will attempt to write down the different problems, solutions and experiences that come up during the course of my internship, for as far as I am allowed to speak of the research that I am working on, since I might not be allowed to disclose everything. Since I will try to get into as much detail as possible, these blog posts could get a bit technical, just a word of warning :)

I will be maintaining two different blogs during my stay in the United States: this one, and a blog that focuses on activities that are not related to my research into facial expressions and gaming. This blog can be found at http://la-pruik.blogspot.com, but that blog will be in Dutch. A link to this blog will be available in the menu to the right.