A talk with research Sabrina Hoppe, who specializes in machine learning and cognitive science. Hoppe was part of a team who worked on a 2018 paper titled Eye movements during everyday behavior predict personality traits. Transcript of this talk is below. We talk about that research and what was found. Topics include: how that study was set up; how strong the correlations were; the possible reasons why there might be relationships between eye movements and personality; and other research in that area.
Episode links:
- Apple podcasts (embedded below)
- Spotify
- Google Play
Resources discussed in this episode or related to the topic:
- The 2018 paper on eye movements and personality by Hoppe et al
- 2015 research by Hoppe et al on eye movement and curiosity
- 2015 paper by Baranes et al about eye movements and curiosity
- 2005 research on eye movement pattern correlation with optimism
- 2012 meta-analysis of research on eye movements and depression and anxiety
- 2021 study on social anxiety and avoidant gaze
TRANSCRIPT
Zach Elwood: Welcome to the People Who Read People podcast with me, Zach Elwood. This is a podcast about better understanding other people and better understanding ourselves. You can learn more about it at www.behavior-podcast.com.
There are quite a few studies that have found connections between people’s eye movements and their personality traits. These studies typically take the form of having subjects wear eye tracking hardware while performing various tasks, and then looking for correlations between the eye movements and the subjects’ rankings on various personality tests.
In this episode, I talk with Sabrina Hoppe; her last name is spelled HOPPE. Sabrina is a researcher who specializes in machine learning and cognitive science. Sabrina has worked on several research projects related to eye movements. A 2018 paper she and others worked on was called Eye movements during everyday behavior predict personality traits, and that’s the one we’ll focus on in this talk.
To quote from the abstract of that paper:
Here we show that eye movements during an everyday task predict aspects of our personality. We tracked eye movements of 42 participants while they ran an errand on a university campus and subsequently assessed their personality traits using well-established questionnaires. Using a state-of-the-art machine learning method and a rich set of features encoding different eye movement characteristics, we were able to reliably predict four of the Big Five personality traits (neuroticism, extraversion, agreeableness, conscientiousness) as well as perceptual curiosity only from eye movements. Further analysis revealed new relations between previously neglected eye movement characteristics and personality. Our findings demonstrate a considerable influence of personality on everyday eye movement control, thereby complementing earlier studies in laboratory settings. Improving automatic recognition and interpretation of human social signals is an important endeavor, enabling innovative design of human–computer systems capable of sensing spontaneous natural user behavior to facilitate efficient interaction and personalization.
It makes some instinctual sense that our personalities, how we experience and interact with the world, would relate to how we move our eyes. Our eyes are how we explore our environment, the things and people around us.
For me personally, I’ve noticed that when I’m more anxious, my eye movements are more restrained, more frozen. And that’s just what I’ve noticed; it wouldn’t surprise me if there were other patterns I haven’t noticed. And it wouldn’t surprise me that some of those ways I move and use my eyes are not just situation or emotion-based, but are part of the way I interact with the world, part of my personality.
When it comes to poker, one important poker tell can be how someone’s eyes move when they’re bluffing compared to when they’re betting a strong hand; often people who are more anxious will tend to be more frozen in general, and this can manifest in how they move their eyes. A player who makes a big bet and whose gaze travels around to various places, who has more dynamic eye movement; that person is likely to be relaxed and have a big hand.
I mention these things just to give some of the specific reasons I’m interested in eye tracking research. And I hope to do more episodes on this topic in future.
Okay, here’s the talk with Sabrina Hoppe.
Zach Elwood: Hi Sabrina, welcome to the show.
Sabrina: Hi, thanks for having me.
Zach Elwood: So you’ve done a good amount of research involving eye tracking and psychology. Was eye tracking something you were interested in early on? And if so, what was your interest in that?
Sabrina: I started my studies actually in computer science, and a bit later I discovered that I was also interested in psychology. And then eye tracking seemed to be a very nice area where I could use my computer science knowledge, but then apply it to psychology because eye movements are so nice to be recorded, and then you can do a lot of data analysis and stuff that this computer science part of me would be interested in. And then of course, additionally, I found that you can have so many interesting questions to answer like looking into concepts like personality, which we will talk about later, and I just thought that this was a really nice area then.
Zach Elwood: Yeah, one thing that struck me looking at the raw data was just the huge amount of data that was gathered by the computer, and it seems like that would be pretty hard to do without computers. Am I getting that right?
Sabrina: Yes, definitely. This type of study wouldn’t have worked without a computer for sure.
Zach Elwood: So maybe you could give a little bit of summary on the existing research that was around before you did the study on eye movements and the correlation with personality. Was it pretty widely accepted beforehand that eye movements were related or correlated with personality states?
Sabrina: Yes. So it was widely accepted that there is a correlation in laboratory settings. Meaning if a person comes in and then you sit them or you let them sit down in an kind of isolated room, no audio distraction, no other people, and they have this one clear task, which would be look at specific images which have been pre-selected to show them, so in this very, very restricted setting, there it was well accepted that there would be a connection between personality and some measurements that you could take from the gaze. But what was new about our study was just taking this to the real world. So we really just sent them go shopping on a campus and maybe they met other people on their way, maybe they had some everyday life things to do on their way, and yet we could still detect this link between gaze and personality, and that was the new part.
Zach Elwood: Do you see your study as strongly contributing to making the case of that link between eye movements and personality or do you think that that was already a respected idea and your work was just part of it?
Sabrina: I think the idea was respected, but we made it stronger because we showed that you can really detect this pretty much no matter what the human is doing or at least it far less matters what the human is doing than we originally thought. So if you think about it, there’s so much that could come up in the real world. Your attention could be somewhere entirely different or you look at something, at other people and so on. And through all of this, we can still detect personality. I think that was a new idea, we really contributed to showing how strong this impact of personality on your gaze is.
Zach Elwood: And if I was correct on my understanding, it wasn’t based on the situation at all, it was just purely an analysis of eye movements, regardless of the situation. They were going out in the real world and going to the shop and such, but the analysis was just purely on how their eyes moved and the traits of how their eyes moved. Is that correct?
Sabrina: Yes, exactly. So we only know kind of in a 2D setting how the eye moved, and we don’t even know what they were looking at. So we don’t make a distinction if that was a face or an object or the way, whatever.
Zach Elwood: Maybe you could talk about the specifics of what you found, what were the most interesting findings to you and how reliable were the findings and such?
Sabrina: So we looked into the big five, which would be five different personality traits. And then out of those five, we could actually detect four in the gaze. That was surprising, I would say, at first, because four out of five would be relatively broad. So it’s not just one specific trait of a human that you can detect, but actually kind of a range of potential personality traits. Plus we also looked into curiosity, which we could also detect, and the performance of our classifier, so of this machine learning method which we applied in the end was above chance. So quite clearly it’s not a finding that would have happened just by kind of random influences on the data, but the performance is not as great as you might expect. So if you just pick a person now and we apply the method on this one person, actually we would achieve at most something like a 50% chance to be right in what we do. However, why this is still interesting is that we took the personality scores for each person and we divided them into three classes. So we had people with a low score, a medium score and a high score. And then kind of chance level if you were guessing would be one third, and we reached something which is up to roughly one half, so it’s above chance level. And that’s basically the exciting part for this data science perspective and just the statistics, but it doesn’t mean that it’s actually super accurate for a single person.
Zach Elwood: So there were three classes of each personality trait. And chance would’ve been a third, but for the ones that predicted well, it was picking those at a 50% chance instead of a 33% chance. Is that basically what was happening?
Sabrina: Yes, exactly. Most of the scores were between 40 and 50%.
Zach Elwood: So what are the specific traits it predicted? Which ones stood out for you as being the most interesting and are you able to say what the actual eye movements were like for those traits?
Sabrina: So the big five that I just mentioned would be neuroticism, extraversion, openness, agreeableness, and contentiousness. And we could detect four, and the one we could not detect is openness. Now, I’m not sure I can make sense of that to be honest, I mean, it’s purely the finding just of our data. It does make sense to me that you can find extraversion, because as a human I would have this intuition that that is something I can see in other people. The way they look at me, I feel like I could judge if they would be willing to talk to new people and you know.
Zach Elwood: Right, they look around more, things like that.
Sabrina: Yeah. So that’s not a very scientific way to approach this, but intuitively to me, it makes sense that you can see extraversion in a gaze movement, in eye movements. On the other hand, somehow that’s also very related to openness actually, and you cannot detect openness. So certainly there would still be lots of room to look into what exactly now makes the difference between extraversion and openness. Why can you detect one but not the other?
Zach Elwood: I know in some machine learning studies there’s a bit of what they call the black box aspect, where you’re not really sure what exactly it’s finding. But I’m wondering if that’s true of this research, are you able to say exactly what it was finding that was correlated with these personality traits? For example, for neuroticism, are you able to say, what was it finding in the eye movements or otherwise that was correlated with neuroticism?
Sabrina: I guess with machine learning approaches, there’s always a little bit of this black box effect, but the method we chose at the time has it less than other methods at least. So normally in machine learning, there is the concept that you have the raw data, that would just be all this data that we recorded as you said. And then we extract something that we call features. So that would be you first analyze the gaze and you try to identify periods of time when the person executes a fixation, which would be gaze at one place for a longer period of time, typically in order to get information from this one place, and then there are saccades in which the person just jumps to the next point, those are like super rapid movements. And so there are different kinds of these movements, you analyze these. And then you extract numbers from them, how long did fixation on average take? What peak velocity did the quick movements reach? I don’t know, hundreds of them really. And afterwards we throw all of these measurements into our machine learning approach. And so at least we know what we extracted. So maybe for some reason because of prior work, for example, you could have a hypothesis that exactly the duration of fixations or something else has particular impact. And then we would be able to verify whether or not this was an important feature in our analysis. But the other way round, so just coming from what the classifier does in the end, it’s really hard to understand entirely what it’s doing, because it’s not based on a single feature, it’s this combination of hundreds of things and then they have some waiting factor and it’s really hard to tell after all. So what is it? It’s not like I as a human could say, “I try to learn this technique now, and I’m going to analyze exactly that.”
Zach Elwood: So is it accurate to say my understanding is for a lot of these machine learning studies, it’s kind of accurate to say that the machine knows something that we can’t know or find out. Is that accurate?
Sabrina: Yes, because we somehow cannot process so many small pieces of information which are seemingly maybe seemingly irrelevant or seemingly unrelated, we can’t process them together as a human, and the machine obviously doesn’t care.
Zach Elwood: If I want to know what the various data was, the specific movements and such related to neuroticism, for example, I can’t really know that, is that basically what you’re saying?
Sabrina: You can do an analysis in the end, so I could give you a ranking of which features were most important. You can know, yes, but it’s still not this one feature. It’s not like the algorithm selects one thing and then says, “That’s it, that makes a neurotic, but it’s always about this combination and that’s hard to grasp, but you could write down the math if you wanted.
Zach Elwood: Have you done that with the various personality traits? Have you shown which eye movements and such and kinds of eye movements were what it was finding in the analysis?
Sabrina: Yes, we did that. But that’s quite hard to decipher as a human.
Zach Elwood: Yeah, it’s pretty hard to… That was the challenge I was getting through. It was hard to find the summary, because like you’re saying, it’s taking into account so many things basically is what you’re saying.
Sabrina: Yes. So if you would check this table in our publication where we did this analysis, a lot of it is about what we call [n-gram] analysis. This would be sequences of eye movements. So if you think we have fixations, blinks, and saccades, then you could combine them in different ways. So it could go blink, saccade, blink or if it was linked three, you can do all sorts of combinations. And then there we just counted them and we analyzed which of the combinations of movements was the most frequent. And now you can say in the end that the most frequent movement which is consisting of three parts somehow seems to be more linked to openness than to others. But that’s very unintuitive I would say. So as a human, you go out of that and you’re like, “Okay, that’s the math, but what does it tell me?” And I think that’s where this black box effect comes in that you talk about. We can write down the math and we can show you what exact analysis we did and then we can score it, and we can tell you this was important, but it’s still not an explanation that a human is…
Zach Elwood: Right. It’s not easy to make sense of.
Sabrina: Exactly.
Zach Elwood: Probably getting into the realm of opinions and conjecture, but do you have opinions about what might be… For example, if we take neuroticism, it seems to be possible to imagine someone being neurotic and having a lot of eye movements or also having very little. For example, one can imagine different patterns existing, one can imagine scanning around for threats and such and being nervous in that way versus someone being kind of overwhelmed by sensory input or threats or whatever, the sense of threat, and kind of shutting down a bit and moving their eyes less. And I’m curious if you have any thoughts on the kinds of nuance and variations that might be involved in these kind of broad personality traits. We sometimes tend to give a general term to things that contain a whole lot of nuance.
Sabrina: Yes, definitely. I think this already starts with what these traits mean. I mean, how did we come up with the fact that there are these five factors in psychology, which is a quite widespread model for personality traits. And still the way that they were defined is basically just by looking at how we describe people. And I think in every language of the world, there are millions of words that you could use to describe another person. And then the research is around the, I think, 1950s or so, they try to figure out which of these descriptions typically come together and then try to identify only a few dimensions in which humans would typically differ. So this is somehow the collection of things which we now look at when we talk about one of these traits like neuroticism which has emerged from these verbal descriptions of people. But of course, it’s still because humans cover such a big variety and neuroticism–
Zach Elwood: And [unintelligible 19.04]
Sabrina: Yes, it’s such a collection of things that just by itself even the category we have there, detecting neuroticism, means all sorts of things. And then additionally, if we picked one, still I believe that it’s entirely possible that different people with this, let’s say, symptom of neuroticism exhibit a different case pattern, because for somebody it leads to slowing down, for the other person speeding up, who knows?
Zach Elwood: Yeah, and you can imagine the same for the PCI which I think is the perceptual curiosity index. And I think they called it perceptual curiosity because kind of in a similar way you can imagine there being different kinds of curiosity. You can imagine there being someone who’s curious about concepts and internal ideas, and that wouldn’t necessarily manifest as someone who seems more visual curious whose eyes move around to different places and such. Would you agree with that, that that also contains a lot of nuance there?
Sabrina: Yes, definitely. You’re right. So curiosity is a very broad term as well, and that’s why psychologists have invented subcategories such as perceptual curiosity. And then still perceptual curiosity could also just be directed towards sounds, and then why would that show up in your gaze? So even there we’re actually only targeting a subcategory, but I’m not aware of a default way to assess only the visual part of perceptual curiosity.
Zach Elwood: We talked about this a little bit, but what were the findings from this specific study that stood out the most to you as being interesting or surprising?
Sabrina: I think the most surprising one to me is just on a very high level that even if people just go out and follow some everyday kind of task, we can still do some prediction about their personality. I think it’s stunning that that works. And then we also checked if we look at the two ways, so the way to the shop and the way from the shop, but we cut out the shopping itself, do these measurements correlate? And then we compare the two situations, so the recording inside of the shop versus the recording on the way there and back. And actually they do correlate which means that even inside the shop or on the way, we’re kind of able to detect the personality traits. Now there are differences, the way there correlates better with the way back than with the time inside the shop. But still it’s there, and you can detect these personality traits sort of independent of this situation. And I think that’s quite interesting to see because there’s so much that changes, just the scene that people look at and you have to use your eyes in order to do the shopping and select your item you want to buy, there would be a person to talk to pay, maybe you meet somebody and so on.
Zach Elwood: Yeah, it is interesting that, and it makes sense instinctually that the way we experience the world would play out in how we look at things. There is an instinctual element to why these findings would exist.
Sabrina: Yes. And there are also studies for other types of how we perceive the world. So I know that there are studies about optimism. So optimists have a significantly different gaze at certain images than pessimist people.
Zach Elwood: Right. If I remember correctly, they were more likely to look away from bad negative images. Am I getting that right?
Sabrina: Yes. So I only recall one study in particular, probably there are more out there. But there is this one study where they presented skin cancer images to optimists and non-optimists, and then the finding was indeed that optimists tend to look less at the negative images while there’s no difference in sort of more neutral face images.
Zach Elwood: And there was another one that stood out to me, it involved doing trivia questions. And people who ranked as more curious, their eyes would look more towards the source, the incoming source of where the answers would come from later. And so there was this correlation between anticipatory eye movement basically and curiosity.
Sabrina: Ah, yes. That’s actually a good example for a well-controlled study. So it’s just participants who look at a specifically designed interface, and it’s designed in such a way that the answer would come from somewhere where there’s nothing else to see. So basically in the end you can really derive something about who was looking there and why would they look there.
Zach Elwood: And you’re saying, because it was so simple and there were so few factors to study, that it really was a well-designed study.
Sabrina: Yes, exactly. And then so maybe it’s a little less surprising that it works, so it seems more obvious that you can detect something. But this type of study makes it much more easy to derive thorough statements in the end. You really know it’s this anticipatory gaze, for instance. Maybe this occurred in our study, but actually we don’t know to be honest, because all sorts of things could have happened.
Zach Elwood: Right. It’s almost like yours is studying everything which includes all these minute things that people can find in these very specific studies, but yours is grouping all these various things together which makes it hard to disentangle.
Sabrina: Yes. And that’s then also this kind of computer science data analysis challenge which I personally like which would be, given this enormous mess of effect, can I still come up with something that would be able to detect one of them in the end?
Zach Elwood: So when it comes to maybe the more practical aspects of these kinds of studies, what do you see the practical aspects as?
Sabrina: We already discussed that actually the performance for you human that you would pick and where you try to detect which personality they may have is relatively low. So therefore I think the main application, if you would call that application, which is [B science], I would hope that it just inspires more people to look into eye movements or understanding humans in general which would just be research, basic research. And then if one day these methods would work better, I think there’s also wide range of applications where you could use them, just we’re not there yet. So this could just be something like smart gadgets that we have. It could be a household robot which is able to detect what personality does my owner have and then react to that. Maybe you need to talk more to an extrovert person as a robot than you would need to talk to somebody who’s less extrovert. Or maybe it’s about teaching scenarios. So if there is a remote tool that students are supposed to be using in order to learn something, then it would be useful probably if the tool could automatically realize what kind of kid is it sitting in front and how could I motivate them in order to learn. Because there, again, maybe you need to talk differently to them depending on their personality. And finally, I think probably there are also opportunities for clinical usage. So being able to understand gaze more holistically I think would also enable us to kind of filter out the relevant information for specific gaze-related symptoms that clinical patients would have. So normally of course, we don’t look at their personality so much, but there I’m more thinking if you record gaze from a patient who has some symptoms which are gaze-related, then it would be nice if you could kind of subtract the impact that their personality has in order to be left with only the signal that you’re actually interested in.
Josh: Hey, quick question for you. Are you someone who wants to be fit healthy and happy? And what if I told you you could get your dream body by simply just listening to a podcast. I’m Josh.
KG: And I’m KG. And we are the host of The Fit, Healthy and Happy podcast.
Josh: Listen, we get it. Fitness isn’t easy, carbs, no carbs–
KG: Just stop, okay. It doesn’t have to be that complicated. And that’s why we made this podcast. We get straight to the facts so you can become your best you.
Josh: So the way to check us out is click the link in the show notes or search Fit, Healthy and Happy podcast on any of the major podcast platforms. We’ll see you soon.
Zach Elwood: Yeah. It makes me think of futuristic therapy scenarios where somebody walks into a therapist’s office and they get scanned in various ways. And it’s almost like it would pick up some pieces of information from their eye movements, some from their tone of voice, maybe some from their actual content. And it just makes me think of these scenarios where all these things combined could be a quicker way to get started and diagnose what people are suffering from theoretically.
Sabrina: Right, yes, I could imagine that as well.
Zach Elwood: Very far off obviously, but just thinking about potential future applications.
Sabrina: Yeah, and slightly scary.
Zach Elwood: Yeah, a little scary.
Sabrina: I’m not sure that’s the kind of setting where I envision myself to be proud of, “Oh, see, look, that’s where it led us.” I guess it would be possible.
Zach Elwood: Yeah. I think there’s a lot of risks of even if it’s done well, those kind of things are exactly the kind of things that make people nervous, the kind of people suffering from those things. So it’s kind of a catch 22, even if it was very well done. Feeling like there’s some machine analyzing you is exactly what many people suffering from delusions think. So there can be some problems with that. So I was curious, with the kinds of correlations found in your study and other eye tracking and personality studies, do you think some of those things it finds are pointing to things that people know kind of instinctually that we may just not know consciously? For example, we sometimes get a sense of how people are and what their personalities are from interacting with them, and maybe some of those things that we’re gathering from people in an instinctual social way are some of the things these studies are finding in a more explicit way. Do you think that’s possible?
Sabrina: Yes, I would agree to that. So I also have the impression that we do have a lot of knowledge just as human very implicitly, in particular about other humans, that’s kind of our daily business dealing with them. And I think looking somebody in the eye or watching other people is a very good example. Humans are really trained to have some impression of another person. It’s like this first impression just within a few seconds you have made so many decisions about the other person. And none of us can put into words how we would do that. It’s actually quite an incredible skill. And we are very, very far away from doing this with something like artificial intelligence or with… So just being able to say what it works like, but then also building something which does the same analysis is near impossible. But I think you’re right that with a lot of studies, and in particular with these more controlled, well designed studies, researchers try to get closer to understanding what exactly it is.
Zach Elwood: I think the uncertainty and the noise in the data, the ambiguity in the data, from the other side it helps explain how badly our human instincts can go wrong too. For example, we sometimes have these mistaken assumptions about other people, these kind of stereotypes. So if someone’s not looking us in the eye or their eyes seem shifty as we call them, we can have the instinct that they’re not trustworthy or such, but it could just be that they’re nervous and they’re completely trustworthy and completely friendly. But just in that moment, these instincts can go bad. So there’s kind of like this flip side. I guess the practical takeaway might be just recognizing how much nuance there is and how much complexity there is in human behavior.
Sabrina: Definitely, yes, you’re right. And there’s also quite a lot of research on machine learning where a system tries to learn something which humans are already able to do or which humans have done in the past, and then it’s actually a problem that this AI if it just learns from human behavior will also learn negative aspects on this human behavior. So for instance, if you’re thinking of building some autonomous hiring tool which some human resource department could use to decide if this is the right candidate or not, if in the past all the decisions that the system sees in order to learn from if they were somehow racist or sexist or have any other kind of bias which is actually unwanted, the machine learning methods are really good at picking them up unfortunately, because they just look at these statistical relations. And if the numbers point to most likely this person would not be hired, then they’re not going to be hired in the future as well.
Zach Elwood: Right, it’s just learning from our patterns. Yeah, exactly. It’s like the studies that show people who have racial minority associated last names are less likely to be hired or whatever it is. It’s going to learn from our bad habits or whatever.
Sabrina: Yes, exactly. So there we need to be really careful because on the one hand, of course, it’s very fascinating what humans can do and how there are amazing people out there who know which person would be good at a job or not. But then if you just collect any data, it’s also quite likely that it inherits flaws that humans have in their judgment.
Zach Elwood: I’m curious, with your work on eye tracking and psychology, do you feel like in your personal life you’ve become a better reader of people? And if so, how has that played out if at all?
Sabrina: So I think the biggest impact was actually thinking about personality or psychology in general, that has definitely affected my personal life as well. But this particular work that we have talked about in doing these data analysis, I don’t think that really had an impact. Because as we discussed before, the types of eye movement, characteristics that we looked at are not very intuitive. So I don’t think any precise knowledge there made it into my personal perception. What does happen more likely is actually that you realize much more about your own gaze. So after spending hours of analyzing where people would look or not look, I did have some episodes in my life where I’m just walking along somewhere and then suddenly I become aware and I’m like, “Oh, see, now you looked at this and that and here, and why did you not look at this stimulus?” That’s more this self-observation maybe.
Zach Elwood: Which is interesting, I think that’s where a lot of ideas for research comes from, it’s examining your own instincts and being like, “Oh, maybe there’s something here.”
Sabrina: True, yes.
Zach Elwood: Yeah. I think a lot about eye movements and gaze just because I’ve suffered from anxiety and depression in my life. So I sometimes have thought about how the eye movements relate to that and how I’ve sometimes felt self-conscious because I felt like my eye movements were restricted and that was an indicator of… My eye movements were very locked down and anxious seeming in the sense that they were more frozen, and I’ve sometimes felt self-conscious of that as an indicator of anxiety. So I’ve had a lot of reasons to think about that and dwell on it maybe a bit more than I should, but yeah, I have had similar experiences of thinking about how those relate to personality.
Sabrina: Yeah. And there are quite some studies which look at links between eye movements and anxiety, so definitely you’re not the only person who had this idea.
Zach Elwood: I should look into that. That’s one area I didn’t look at at all. Any other things that we haven’t touched on that you think would be interesting to this audience maybe about that study?
Sabrina: One thing I found quite interesting is actually also on the data analysis side, and that is that I told you before that we discriminated people into three classes. So for each of the scales that we looked at, there would be people with low scores, medium, and high scores. And then actually one thing you always have to keep in mind with these studies is that in the world’s population, kind of medium personality traits are most common, far most common. And so actually these classes are designed such that they roughly have the same number of samples, but this actually also means that the middle class is really narrow in comparison to relatively large ones for low and high scores. And somehow this is an inherent effect because how would you recruit people with a really extreme personality? It’s very hard to get. But I think it would be interesting if somebody managed to get these participants to look at what really extreme personality leads to in terms of eye movements.
Zach Elwood: Do you want to share about other… Are you working on other work related to eye movements or other psychology work in general?
Sabrina: No, unfortunately, no. I changed a bit. So my field of research at the moment is more in robotics and data analysis, so kind of back to my computer science roots. So I have to say no, but I think my co-authors, all of them, are still in the field. So definitely there’s a lot of interesting work going on.
Zach Elwood: So I was curious, it seemed like you had 42 maybe subjects in the study, and I could just be pretty ignorant on scientific studies, but sometimes I think about these studies is that the sample size seems small, but am I just off base on that? And maybe you could talk a little bit about the sample size and why that’s an okay number to get a statistically significant finding from.
Sabrina: So on the one hand, I think your iteration is entirely right, that more people would’ve been better in any case. But this is probably true for every single study ever. As long as it’s data-driven, the more samples you have, the more reliably you can detect something. And then also the more subjects we have, the more we actually cover the scales of personality. But the other aspect to consider is just practicality. So you really need to find these people, they need to volunteer to participate in your study, and then they come in and you need to set up the eye tracking machinery, and then they need to fulfill the task, they need to go through some questionnaires. So it really just causes hours and hours of work. And so in practice, you need to find a balance somewhere. Now, if you wanted to do one of these restricted studies in the lab, there are actually statistical analysis that you can do beforehand. So if you have particular assumptions like let’s say I assume the personality score is actually following a Gauss distribution and I want to detect some hypothesis with a particular significance level, so I can set this to some percentage and I expect the effect to be of a certain size. In this particular setting, you can actually do the math and you would get a number. You need at least, I don’t know, 50 people in order to detect your hypotheses with 90% probability if the hypothesis is true. So this exists, but in our case, since it’s one of the more exploratory, just sending people out into the world and then we apply machine learning, which is a slightly different setting, this is not possible or I’m not aware of how you could derive this number mathematically. So basically what we do is just looking at previous studies, how many participants that they have, and then what’s our experience with past studies we did, so we somehow set the number.
Zach Elwood: And for your study, if I was understanding it correctly, you were actually doing multiple runs where you would choose one set of the subjects as the analysis group and then another set of the subjects as the ones that you would try to predict for. Was I getting that right? That basically you were making multiple runs to both train the algorithm and make predictions for the remaining subjects?
Sabrina: Yes. So the way we train the algorithm in computer science is called cross-validation. And this means that we divide all the data that we have, and in particular the people that we have into these different sets. But if you only said, “Okay, I take, let’s say, the first 30 people as my training set, I train my algorithm on this data, and then I evaluate on the other 10,” you would actually run into trouble if these 10%, if these 10 people in your test set are somehow not representative. So I mean, by chance maybe no person with high neuroticism is among these 10 people, so to correct for this effect, what you do is you do multiple of these splits. So let’s say once you really take the first 30 people for training and you evaluate on other people, and then in the next round, you actually take people 10 to 40 and you would test on the first 10 and so on. So you shuffle the data multiple times.
Zach Elwood: That’s great. I think we’ve covered a lot. Is there anything else you want to mention about work you’re doing or things you’re excited about?
Sabrina: No, actually I can’t think of something right now.
Zach Elwood: Well, thanks a lot Sabrina for coming on. This has been very interesting and thanks for taking the time.
Sabrina: Yeah, thank you.
Zach Elwood: That was Sabrina Hoppe. You can learn more about Sabrina’s research by looking at her Google Scholar page.
If you’d like to see some of the studies discussed in this talk, go to my site behavior-podcast and look at this episode’s entry.
This has been the People Who Read People podcast, with me, Zach Elwood. If you like this podcast, please leave it a review on Apple Podcasts. That would be hugely appreciated.
Thanks for listening.