Name: Human–AI Co-evolution Principles and Examples
Uploaded: 2025-06-17T19:26:04.467Z
Duration: 1 h 3 min 16 s
Description: Human–AI Co-evolution Principles and Examples

Transcript for "Human–AI Co-evolution Principles and Examples": Yes. Alright. And we're live. Hello, everyone, and welcome to yet another Cosmos community webinar. Today, we have a fantastic, really, really interesting topic. And we've got, doctor Huang, Wang here with us, who is a research assistant professor at the Center for Human Artificial Intelligence and Robot Teaming under the Global Security Initiative at Arizona State University. She's also the founding chair of the human AI robot teaming technical group at the Human Factors and Ergonomics Society. She has worked on ARL, ONR, and DARPA research projects as a research lead. And here today, she's gonna talk about human AI coevolution principles and examples, and I am super, super stoked about this. If you have any questions for doctor Wang here, please put them in the q and a. We'll handle the q and a after the presentation. So with that, doctor Wang, take it away. Okay. Thank you for having me here. I will share the screen. Hi, everyone. Thank you for joining the webinar. My name is Michelle Huang. I work in a center for human AI robot teaming at the Arizona State University. So, a a brief claim that, all the views and positions are mine and not, necessarily represent to the funding agency of Dauphin army. And so, my background is in human factors and applied cognition. So I bet it may be very different from a lot of members in the cosmos. So, before we get started and, have a quick, survey of what is your background, to, to help me understand who are the audiences are. So if you can scan the QR code and enter your background, that can help me, view. I will have a thirty second, time with you. Okay. Alright. We may not have many participants today, but that's okay. We'll continue our talk. Let me share the main screen. Yeah. So today's agenda, I will briefly cover the center that I work at and then dive into the main topic of human AI collaboration and then, talk about, how the work, how your work related to the topic we are talking about today, and then leave the time about fifteen minutes for the q and a. And so first, the Center for Human AI Robot Teaming, it is a multidisciplinary collaboration, unit and the university it's a independent unit that work across campus with computer science, robotics, laws, and art and social science, all these different disciplines to work together to revolutionize how we ensure national security through human machine teaming. And we provide a team science, metrics, measures, and test beds, robotics, AI research and trust measures and human fact methodology, human computer interaction methods, for this research. And this is the, chart leadership. We have, Jamie Gorman, Nancy Kuer, they both are leaders in teams research and spring in mechanical engineering and robots, and to me are in human factors, and we have staff support. We have more than 70, affiliates in different discipline to work on complex problems that each can contribute with their expertise, to work on these multi million grants proposals. So that's our strength and we look into the real world task problems and to identify the key features, challenges in the real world task, and then build, test beds in the lab to simulate the in scenario in the real world and then design a human subject experiments and collect data to test whether the design of certain features will solve that problem or prove to be better than the others. We use a, ghost lab to simulate Mars to study the latency in communication. And we have heterogeneous robotics teams always, Husky. There's Yumi and there is also, Fetch and some Burger bots. A number of these robots we also use the Minecraft a lot which I will talk more later and also air traffic control simulator and remotely piloted aircraft simulation in distributed areas. And, these are the resources we have, for collaboration if people are interested with the test bed measures and studies. And then, we'll dive into the the main topic today for human AI coevolution. And for here, I also have a quick question about what is the what words come to mind when you hear human AI, evolution coevolution? So we also have, like, thirty seconds if anyone, can answer. Okay. Thank you. Yeah. Let me switch back. So recently, I presented, my talk, most of the contents in this talk to the human AI co evolution workshop in Singapore, at the iClear workshop. IClear is one of these, leading computer science conference, and they are also interested in this topic. And in their vision, they aim to address the critical aspects of humans and AI systems, work to work together effectively as AI continue to evolve. How we understand the dynamics of human AI interaction and ethical implications and the practical applications are critical to enhance our daily lives and professional environments. So then, while people invest to millions of dollars on a large venture model and the AGI and all these, areas. Where are we going? Like, what is the ultimate goal for building the AI, system, and what is the ultimate goal for human AI coevolution? So for this, I have one quick question about, this. So what is your stand for the future of AI evolution? So a is AI should eventually be created to function like humans capable of doing everything we can and be granted rights equal to humans and second is AI should be developed to serve humanity even if their capabilities exceed ours in certain areas. And, also, I undecided. Need more time to reflect. Okay. So I I see, several people, on two, two people on two and one on one. Let me switch. Yeah. So from my stand, this is, like, my view that in the future, we try to develop the AI and co Evolu to eventually to promote the human, flourishing. So this is a group of religious leaders, from basically, from Christianity background. They are, like, back in 2019, I was surprised that these group of people coming together and, brought their ethical principles for the future of AI. And for the Ryan and Dessie, they are famous in the self determination theory, and their ultimate goal for any type of the technology development is for psychological well-being, especially the your day money or realization of one's true potential and personal growth. And for me, I agree with this, strength, this trend of development of AI evolution is promote human well-being. And I will use the human emotional attachment model to illustrate how this can be achieved by, our design and technology work and research to achieve that. And so for definition of evolution, from the dictionary it says it's a process of continuous change for lower simpler or worse to a higher, more complex, and better state. So if we break down, this topic, it could look at the how individuals evolve, in in their lifespan or generations and then how AI systems evolve through the advancement of software, hardware, and user interfaces, and interactions, and cross product life cycles and generations, and how human AI influence, each other and advance together through the interaction mutual adaptation. So for these topics, it might be a little bit, vague. So I will use this, interstellar clip to illustrate, like, in the future. This is, like, one of these ideal scenarios that human AI can work together to, solve problems effectively. I will, this is four minutes and fifth fifteen seconds. There is a moment. Just checking, you can hear sound. Right? It is not Cooper, there's no point in using our fuel to analyze the endurance of spin. Cooper, what do you do? Docking. Endurance rotation is 67, 68 RPM. K. Get ready to just to pause that this this is a case and the tires, they are embodied the AI system. K. You're ready to match up spam with the retro thrusters. It's not possible. No. It's necessary. Endurance City Stratosphere. She's got no heat shield. Ace, you ready? Ready. Cooper, this is no time for caution. Ace, if I black out, you take a stick. Stars, get ready to engage the docking mechanism. The turns are starting to heat. 20 feet out. I need three degrees starboard, Cooper. 10 feet out. Cooper, we are lined up. Initiating spin. Come on, Charles. Come on, Charles. Retro truckers. Green Ninja dot. What you got about it? Come on, baby. Killing main engine. Okay. And for our next trick It would have to be good. Red in the Gargantua's pool. So this is a, extreme, example in a very dangerous complex situations that, two, AI, embody the AI system, CASE and TARS, working together with these, astronauts to work, to complete a really difficult task. So, in spite of by this, scenario and so in in the lab, we There is a moment. So, in the lab, we try to design some sort of AI system that is able to observe the environment and then represent what is the problem space and then infer what's the human's goals and beliefs and predict their actions. And then they can intervene through some sort of, modality and then improve the team performance. So that's the the five year, DARPA sponsored, assist program. And then we used Minecraft as a, task scenario where there's different roles and, different skills, and they work together to achieve a, search rescue task. This is a three minute explanation of how we design a human AI team, together and to solve a complex problem. And then, I will explain after the video. Project or ASSIST for short. A building is structurally damaged after an unspecified incident. Injured victims are trapped and need assistance. In this simulated environment, the team has to locate, stabilize, and transport both regular and critical victims to safe zones according to their injury type. The team consists of three human players along with the adviser. Each player has unique roles, knowledge, and skills. The engineer removes rubble and knows the location of hazardous rooms, but moves slowly. The medic diagnoses and stabilizes victims with a normal walking pace. The transporter can detect victims and moves the fastest. All of the players can transport stabilized victims. Their goal is to work together to search for and rescue as many victims as possible. The adviser can infer the mental states of human team members, predict their next actions, and intervene with advice to improve their teamwork. To develop the best ASI advisor, we created eight different conditions. The human advisor can use voice to communicate, while the ASI uses text chat in Minecraft. Each quadrant here represents one of the computer interfaces. The top left is the administrator's top down view of the building. The upper right is the medic in red. Bottom left is transporter in green, and bottom right is engineer in blue. They're equipped with a set of marker blocks on their toolbar and have a real time dynamic map showing their position as well as marker placements. In addition, unique information on the bottom of their client map can help them find the rooms that have the highest concentration of critically injured victims. Here's an example of the human adviser giving advice. This is your adviser. There seems to be miscommunication. Which c zone needs to be cleared? Oh, yes. So it's South South Zone C. And here's an example of the ASI adviser giving advice. Let's see. We ran a 113 valid teams involving 339 participants nationwide. Participants joined remotely through Parsec to play on our lab computers and used Zoom to verbally communicate with their teammates. This study resulted in one terabyte of data accessible on the ASU library data verse. The test bed itself is open source and on GitLab. Data analysis is ongoing. Here's a list of some of the measures we used in this study. Here's an example of IHMC using a subject matter expert rating process to analyze good and bad teamwork. The visualization depicts the player's individual and joint actions, and University of Central Florida analyzed how team profiles predict team performance. We would like to acknowledge the program sponsor DARPA as well as all of the performers who contributed to this project. Thank you. So this is the one example of search rescue, and you can see that, machine readable text is, is is one way that we can inform them. They they are not as advanced as using voice natural language yet as you see in the movie. And next to to test the generalizability of the artificial social intelligence, then we designed a different study with a bomb disposal game. So in in this case, there's a three different type of bombs, and these are standard bombs, and they each have a different color wire. And you have to purchase the correct wire piping tool from the store, and then you will need to have two people coordinate to cut the wire in the correct sequence. If you cut it incorrectly, then the bomb will explode as fire and the injury. And the AI monitors the human behavior, on the test set based on the time stamp when he is at the right, on, how he should strategize. And we also have the in game intervention messages, and the players can respond. It is not as natural as auto natural language, but it is one way that machine can pick up intervention method. And on the map on the left, it shows the self location. Next. Yeah. So, the two, study three and study four, the search rescue bomb disposal, they are the AI is serving as a adviser. Like, they, give you advice on how you improve. And then, Voyager developed by NVIDIA's team, They started to, search the Internet of knowledge to build a, Minecraft agent that can do all the constructions just by processing all the knowledge on the Internet. The videos, on YouTube and then the Wikipedia page, all the blogs, all the website. And then they develop a skill library. The Voyager can do all these things by itself. But, in the real world, we still need the humans in the loop. So how can we develop a agent that can work alongside with humans? And that, leads to our, extended project, also sponsored by DARPA called Adapta two. And in this game, we developed the a TARS, inspired by the Interstellar movie. Tars, has this blockage, shape and is exactly look like the Minecraft characters. And here is a brief introduction of the Tars. This tower defense scenario challenges players to defend their base from waves of various mobs over multiple rounds. Players must mine gold, upgrade their weapons, and purchase fortifications to withstand and defeat the incoming threats. Each round in the game has two phases. In the planning phase, you can devise a strategy to prepare for the coming battle. Once you're ready, click the ready button on the column in the center of the base. Remember, no mobs spawn during this phase. Next comes the action phase. Here's where the gold spawns and waves of mobs begin to attack. Fight off mobs, mine gold, and fortify your base. Each round can have multiple waves of mobs, and there are a total of 10 rounds to survive. A number of items can be purchased using gold. Right click on the merchant in order to open up the shop window and hover over the items to see their attributes. Your weapons and tools can be upgraded, and wall fortifications can be acquired here. Here's a table of the items available to you. Please pause the video if you'd like more time to review. That is five blocks wide, two deep, and two tall. Each of these adds 20 points to your fortification bar, which acts as a shield to protect your base health. Walls can be damaged directly when enemies stand within range. In addition, when mobs get within your base's boundary, which is marked by a blue line, your fortifications will take damage first. Once they are depleted, base health goes down. There are three types of enemies that you will encounter attacking your base. Zombies are slow, do the least amount of damage per second, and only have melee attacks. Skeletons have moderate movement speed, deal slightly more damage than zombies, and can use their bows to deliver ranged attacks. Spiders can jump, move the quickest, and their melee attacks deal the most damage. Here are some general tips to help you in your task. So, this is general, technique training, and then here is the interface that you can, use a large language model, interface to negotiate with the AI agent to talk about what's the best strategy, how they plan for the next round. And then the, AI agent, TARS, can implement all these code into actions and and to kill the mobs and mine gold with you in the environment. For this scenario, you will be paired with an AI teammate. You can interface with this teammate via the left hand side of your screen. Type your message in the prompt field and press enter. This natural language communication can be used to plan and discuss strategy. The agent can help you with tasks like mining gold, upgrading tools, and killing mobs. When you have finished communicating and are ready to leave the planning phase, select the send plan to agent button near the bottom. Now, back in Minecraft, you will see a message that the agent is preparing. Once it informs you that it's ready to go, you may hit the ready button in the middle of your base and begin the round. During the round, you may also communicate with the agent via the Minecraft chat box. However, responses will be more limited. Thank you, and good luck in your mission. Yeah. With, these, studies, we generated, five big datasets, from from individual to teams, from infer prediction to intervene, and from advisor role to, like, a task or performer role. And, all these data except the study, ADEPT two, we haven't, published them. But the other ones are all available on our ACU data repository. And analyzing these data, like, complex data dynamically requires a lot of, like, a multimodal data analysis skills. And as you see that the AI agent could function in different roles, such as a decision aid to help you make a decisions like, chat GPT sometimes, can, can help you calculate and then, detect some of the, autonomous, web target recognition system then help you identify some of the tasks, targets. And then the AI can be a or the visor role as we see in the assist program. It can also be a task performer that actually carry out the physical tasks alongside with you. So these are all possible ways that we can, develop the AI to do. And I wanna say that the, effectiveness, what which we observed from all these tasks, in the build, or some task scenario is one of the attributes of the AI. And, as we see that these attributes of any, AI or human or nonhuman or pets or these, entities, their features will be first, like, filtered or perceived by the individual, which is us. We have a self-concept self worth that that represents our knowledge, skill, desire, and the preference. And then if it's, the attributes congruent with the self-concept, then it will lead to emotional, make a positive emotion, as, a indicator. And then if it's, incongruent, then people will be indifferent or detached emotions towards the entity, and that is a continuum from extreme no interest to extremely interested. And then for the positive end of emotional attachment, then it will lead to more interaction behaviors like, more interactions or talking about it or, like, learning about it to all kinds of things. And then this interaction behavior will lead to the self development of the self. Then you learn new things, and you can capable of doing more things. And then you have the, option of for choosing what you would like to interact with. And then the same attributes of the entity may not be perceived as same after you interact with it because you upgraded yourself. So this is a model that, represents how human grow from infant stage, that doesn't have infant self-concept to, to to elderly going to grave. Like, that is the lifetime learning process. We interact with things and learn and and then upgrade. And so, there's a self-concept, self worth is the core mechanism of developing human emotional attachment, which has this human psychological needs for autonomy, like, your option, your desire, and then, competence is what you can do. And then, related to the needs is how you connected to the world. And this model can be applied to, all kinds of different situations. And this is a one example of applying the model to robotics tournaments that these are the attributes of robots, more features possibly, and what kind of the functions, the forms, whether you own the robots, whether it's unique, and, whether there's a robot related people that you like to interact with, or there's a robot related incentives. These are all attributes of the robot. And then there is, the self, the unique self self-concept. It's also a self regulating system that, consists of your, personality, your motivation, your, the the meaning of the robot to you, and there's, identity, knowledge, skill, career into self, image or the future self. This is, like, everything about you. And then if it's congruent, then you'll it will lead to attachment emotions and which can be measured through all these, like, trust to feeling close or feeling attached. All these emotions can be measured. And then there is also, interaction behaviors in terms of general, interactions. And then when it is separated or damaged, the, people feel miss, like, we'll miss them, repair, or videotape, write about it, or all kinds of things about it and, indicates the human's emotional attachment to the entity. These interactions and learning process also, contribute to the personal growth in the process. And, if these are facilitated and internalized, then human become intrinsically motivated interacting with the entity and will lead to better performance, better learning, and more satisfaction and, more, better human well-being. And so, one thing I went to the the recently, the, sent to for human compatible AI, they have a workshop try. I I went and, I heard of one concern from the AI developer's perspective is that they're concerned that the human states are unmeasurable. So you cannot easily use, like, reinforcement learning or some other algorithm to train the AI. But, from human factors perspective, all these human states, either observable or latent, are measurable through the surveys, through the communication, through their behavior, through the physical measures. They all can be measured, and then the model can be applied to any type of, human AI robot scenarios. And so, then how do we achieve this human AI co evolution? My position that we need to, like, to co train, co learn, like, from humans, Once the AI features, the attributes are developed, you need to learn, about it enough to understand how to use it appropriately to achieve calibrated trust and optimal adaptability, all kinds of scenarios, troubleshoot them when it goes wrong. And then by then, you can learn, to establish appropriate level of operations and trust. And then on the AI side, which, is not my background, like, how AI learn about humans, challenges, how the individual differences that you need to consider, and especially on human subjects of studies that we, our dataset has a thousand trials, but in machine learning field, that's still a very small dataset. So how can the, machine learning engineers and, data scientist, they can utilize a smaller data set to train AI to be targeted, in certain behavior attributes. That is a challenge, because, getting data, is is expensive. So there's, how you can train the AI to learn from humans' preferences, behaviors. And then for, part three is, like, any functional AI eventually involves humans. Even though there's a singularity theory talking about AI will exceed far exceed human capacity and they try to do self driving cars. But self driving cars, still you have passengers, and you also have dispatcher center to monitor which car goes wrong and which car needs the maintenance, engineers to come, onto the rescue on the road. So there's somewhere, like, an even though people try to design human out of the loop, but somewhere human has to be involved either to monitor or, like, work alone or, like, use the services. So there's always consideration for humans. And we, we, learn from the real world task and challenges, identify the problems, and then we design the algorithm to to try to solve this problem. And then we'll have to use a somewhat of user interface either by graphic voice or touch or command line, any type of interface. And then there's a interaction with the AI. And then we have, we it's essential to evaluate to how well this human AI is working together in this in this real world. And then, for human factors, like, what we can do in the whole process, we can analyze the user needs through their like, what is the biggest challenge, in their work in their task procedure. And we can identify these challenges and problems and then, have a targeted design for solutions and prototypes. And then we can work with, AI and, developers, to figure out what is the key feature requirements for the product service to work. And then, what is the appropriate interface for users to understand what is going on. And then we evaluate, by collecting data, in designing experimental conditions to train humans to understand how to use the system and then how well they are using the system. And then using this, statistical dynamical model methods to analyze and provide recommendations to further improve the AI system. So, and as I said in the, attachment model, different people will have different preference and different scenario. They will have different, priority of the features. So that's where we can study the typical user group. They, as a group, how do they prioritize the features? How do do they typically, like, the conduct the task procedure? What is most likely to incur errors? And these are all the, problems that human factors can use our methods to identify. So for your research, I know, I don't know where exactly you are, but most likely in the AI algorithm or data analysis part, but we can all work together, on this. So if you have, like, a specific, like, a use case scenario that we I can, I offer thirty minute free consulting on AI technology design and how you possibly, can design some tools or procedure to understand or better design the user experience in order to achieve human well-being? So I put the contact information and the booking, page on the site. And these are the collaborators, contributors, and the funding agency for the related work that I presented today. So next, we have time for questions if any. Yes. We, and so I have a few questions for you. The the first one is thank you so much for for the presentation. That's not a question. That's a comment. But thank you so much for the presentation. It's really, really, cool to see. And I play a lot of Minecraft with my daughter, so I'm definitely gonna dig into this a bit more and see, see how we can use this here, just to play around with because this is super, super interesting. Before we started the presentation, you and I, spoke a bit about the emotional attachment to, robots and AI. And I mentioned that I say thank you, to AI when I use it. But then you came up you, talked about a, a really interesting, experiment where people were dismantling robots and how attached they got to them. Do you wanna, expand on that a bit? Yeah. So the reason, like, as I reflect on why these people would, feel sad about dismantling is because they spend a whole semester coming up with an idea about the features of the robots, and then they learned, through the procedure about how to, implement the features. And then the robot is eventually functioning. So there's a whole process they project to the their self, the concept of self into the robot. They they have this, as a the self determination theory said, the human has the basic three basic technological needs. The need for autonomy is is the free choice, of of the, sin situation. So you you design which feature you want. You have the option, and then there's the the need for competence. They learned and they grow. And then, and then there's a need for relatedness. All these teammates working together, the teachers, working with them, they are all supporting each other. So through this process, the human psychological needs were met. So that is congruent with the self-concept, self worth. And by these features, then they would, inform emotional attachment with the robot because they spend time. This is memory. This is part of who they are. And I when I ask them what does the robot mean to you, and some of them, they say it is assistance because they help me do some certain things. Some of them say they it's a friend. I talked to the robot as I built. And they someone say they said it's my baby because I bring it to life. So for these, like, they in they project a part of themselves. It's a embodied, self extension to remember who they are and, being part of their time and experience. So that's, is aligned with the mechanism of the emotional attachment formation. And that emotional attachment you mentioned, you don't want that all the time. Sometimes you wanna use robots for for certain tasks that where you don't want to have an emotional attachment to them. Yeah. So as I mentioned, there's a book and also news reports about soldiers get so attached to the robots, they hesitate to send the robots to the field to be destroyed. And then they after the, after the robots are destroyed, they have 21 gun, ceremony to, to celebrate the life of the robot or they bury the robot, with a very formal ceremony. And then so for these, their grief is, and also the attachment is hindering their correct decision making. So that is something we don't want, and or it is not good for the mission success in some sense. And we will need to, use some, like, counseling or some training to reduce their attachment, if that is to certain degree, it's not a healthy attachment. So we'll it's not always good to have a debt attachment to certain, AI system. Like, people may get addicted to TikTok or some other, like, videos, like, feeds, because it is interesting. And but, being addicted to things for too long without self control, it is detrimental to the mental health. So we want to, regulate certain characters or the attributes of the AI system so that it is I mean, it has advanced features but should be beneficial to the human's well-being. So humans can, learn to regulate themselves, but as AI researchers or AI, governance, policymakers, they also need to regulate the AI before it gets out of control. I think that's a good idea as well. When we look at your research, we, you you span emotional states, behavioral patterns, cognitive processes. So how do you measure and model something as complex as emotional attachments that we that we just talked about intrinsic motivation in these interactions. How do you how do you measure what kind of attachment and what kind of motivation, or how much is being applied here? Yeah. So, I did the initial study in robotics tournaments, and I went to, many of these robotics tournaments and interviewed more than 200 teams, and I'm asking them, to fill out my service. And so from the surveys, I ask them, like, questions. For example, from, one to a 100, how much do you like the robots? And then explain the reason why you make that choice. And then I ask for the, the what is the meaning of the robot to you? And then, we'll also have questions related to to how well this, robot is, satisfying the requirements of their, like, help them feel competent or help them feel connected to these people who are trustworthy and supporting. So the the first stage of the study is using surveys. But then, as I work at the Center for Human AI Roar teaming now, we focus on nonobtrusive, like, measures. Like, surveys may be interrupting to your task or procedure. You don't want people to just stop and do your survey and then continue the task. So we, propose to study team interaction by their, communication. Like, when they say things, do they, based on what they naturally already been doing, can you identify meaningful patterns from their communication? Do they say positive things? Say, like, a beneficial or, like, being, grateful or these these things, you can identify certain, patterns from the communication. And then you can also identify certain behavioral patterns. If there's a AI giving you advice or working alongside with you, would you choose to work with the AI by taking the advice, or you will totally ignore the other device? So if you are ignoring the other device or shut it down or, like, not responding to the request, these are the behavior that indicates that you don't care or you, like, dislike this AI system. So these are, like, a behavioral or nonobotrusive indicator for your detachment of this AI system. So to, add on to that, how do you, how do you address the challenge of designing for different human responses to AI? Like, how do you, how do you make it user friendly to as many people as possible where we have vastly different mental models of AI capabilities and, AI functionality? Yeah. So, it will all comes to who is your audience. Like, if you have a, like, if you can find the demographic information about the typical user group, then identify what is the biggest challenge they have. And then, like, a either it's a small or, like, if if it's a smaller, then you you can do a folks group or have a usability testing of or sync cloud to see them walk through the step by step and identify where is the error happening. Where do they get confused? Where do they, like, making mistakes? So these will be, representative data to troubleshoot where the problem is. And if you have, like, a hugely large, audience, then it is probably you like, in our study for design because we aim to collect more than a thousand trials, which is, like, abnormal in human self study. But, it's delta program. It's ambitious goal. So we designed the a adamant this, test bed that participants will come and wait in the, waiting room. And then, first in first out, three person can spin up a test by automatically. So, if you have, like, a huge number of, of audience, then and if you want to see their data, then, use some kind of automatic procedure to identify the features, what you care the most. And this will based on the stakeholders, the interest and also the subject matters, opinions on depending on which of the behavior you care the most. If, like, for example, if if it's a shopping, center, website, you care about how people view the I, select the items based on the recommender system, whether they actually put that item in the shopping cart, whether they actually check it out, how long they check this thing out, again, which page is causing the problem. So each of the scenario, you can rely on this, human subject experts to identify what is the most relevant feature that my users will, face the most frequently, then I, we can, we can use if if the website already log automatically log some of these behaviors, you can pull this information and analyze, and, and also analyze the cost for each of the feature, to resolve them and then prioritize. And then once you, implement a prototype of the feature, it is normally beneficial to have user testing before you, like, implement the thing to the entire, like, system. So a smaller scale of testing whether this feature is, working as desired is is, cost saving and time saving. And, normally, companies would have UX developers, engineers. They can do this kind of job. So yeah. So always go back to what is the greatest challenges, problems of your typical users. And then, of course, you need to identify who are your users. Not everybody. Not not in any, like, range, but you will have an idea who are the typical ones. And then work out the task or procedure, see which step, which interface, which of the information is causing the problem. And then we, work on the prototype and then test the prototype solution, see whether it improves the performance, and then, we implement it. Awesome. I got one last question for you. So given your vision of human and AI coevolution here, how do you see the role of humans, us as professionals here changing as AI systems become more sophisticated? What new skills or perspectives will be most critical for this field in the next decade, do you think? Yeah. So I think the humans are relatively stable, like, from generations. But also in the past one hundred years, humans also, like, the knowledge, skill increment is, like, significantly faster than the previous generations. So because of the humans' adaptability, and this, cable and this technology allow us to accumulate this, skill knowledge, we can learn, some of these necessary informations like, how to use the tool. Not everybody needs to program all the AI tools, but you can learn, to how to apply these AI tools. And for those and identify who, like, humans not only need to learn about the AI, also need to learn about themselves, what I'm good at, like, what I, like to do, what what is my expertise, like, what is my experience and capabilities. And then based on that, you will be able to find something that you are good at doing. And so based on that, you you learn how to utilize the AI tools to help you do something that yourself cannot do. Yeah. So this is like a human evolve by utilizing new skills, pick up a new knowledge, and then you can improve, to become a better version of you. And that is the path of human development in a lifespan. So there's always something that you can explore, that you can something that you can experience or something you can you can do well. So it's it's ongoing from from crab or infant to grave. Now they go lifelong learning process. Yeah. And from the AI, they need to understand the users and also learn to be adaptive. Like, the the users is very the VAST Data users are very different. Each individual have different personality, behaviors, and the preferences. And and so, then by statistics, what is the majority of their preference and, and their, behavioral patterns? And then how can we train the AI system to be adaptive to most of the participants or users. We cannot accommodate everybody, and, I mean, at a reasonable cost. Yeah. But, typically, the way is to identify meaningful patterns that represent the most of the pit typical user group. Yeah. That's how we can achieve, bidirectional human AI coevolution. I think you you hit the nail on that there. I I think this is wonderful. I wanna thank you so much, doctor Huang, for, presenting this to us today. This has been fantastic. Super, super interesting. We'll be posting this up on YouTube, most likely tomorrow, and then we'll have a, summary as well posted on the VAST Data community over at community.vastdata.com. Perfect. I was just going to ask you, if you wanna put up a contact details there. Do you have any closing words for us? Yeah. I'm super excited to to see, the community interested in this topic. And, I look forward to, in the future, that human factors can help the, AI community to to develop better AI agents, teammates, and, will will eventually lead to human, psychological well-being. That's that's the goal. Fantastic. Fantastic. Thank you so much again. And, for everyone who's been watching and participating here, thank you all. And, we'll see you at the next webinar. Have a great week. Thank you. Bye.