In this episode, I speak with Shan Luo, Professor and Chair of Robotics and AI at King’s College London, about the relationship between a robot’s sense of touch and the models that learn from it.
Shan is one of the professors whose name was mentioned frequently when I speak with robotics and tactile researchers. He has chaired ViTac workshpos for X consenstuive years. So he’s seen the tactile industry evolve close-up.
The question that stuck with me was about replacement skin. A tactile sensor’s soft contact layer wears with use. Replacing it can change the material properties and alignment that a force-prediction model depends on. Shan’s GenForce research helps transfer knowledge learned with one sensor to another, reducing the need to collect force-calibration data again.
We also discuss his work with Unilever on assessing hair and fabrics, the force-control problems he describes with Ocado, and how the task should shape the finger, sensor and control policy together.
There is an Easter egg near the end: I ask Shan what he would build if he started a company. His answer includes a surprise announcement. I’ll let him tell you.
Topics:
The contact information behind material assessment and manipulation
GenForce and learning across different tactile sensors
ViTacGen, object pushing and learning-time touch
Replacement skin and the challenge of preserving calibration
Designing the sensor, finger and control policy together
Industrial research with Unilever and Ocado
Open datasets, shared benchmarks and a surprise announcement
GenForce research · ViTacGen research
Shan Luo · King’s College London profile
Watch the conversation:
Explore more conversations on The Core Matter Show.
Recorded remotely on August 21, 2026. Published October 2, 2026.
Chapters
00:00 Cold open: what happens when robot skin changes?
00:29 Why Shan Luo chose tactile sensing
02:04 Scaling tactile data and simulation
03:40 The interface between different sensors
05:04 Material assessment, cleaning and insertion
09:18 Vision, contact and the last mile
11:54 Manipulating transparent objects
14:29 GenForce: transferring force-prediction knowledge
17:59 Hardware, shared datasets and standards
21:43 ViTacGen: learning from touch, deploying with vision
25:42 How tactile sensor skin wears
28:43 Replacement skin and model calibration
30:49 Manufacturing cost and consistency
32:38 TacDiff: designing the finger and policy together
36:47 Industrial research with Unilever and Ocado
40:01 ViTac workshop and open tactile data
45:11 A startup question and a surprise announcement
49:43 What warehouse robots could feel in 2031
Transcript
00:00 · Shan Luo
But the real question is, can we make sure the model can still work if we change the skin? The group has to keep collecting essentially the same kinds of physical interaction again and again. That’s why at deployment for some tasks, the knowledge can be carried by vision without requiring the tactile hardware to be present. Thank you for the invitation. I’m very, very happy to join the podcast.
00:29 · Michelle Sun
So you did your PhD at King’s, and then you went to MIT, Leeds, Harvard, Liverpool, and you’re now back at King’s holding the chair. When did touch become the thing that you want to spend the next 20 years on?
00:47 · Shan Luo
my interest when I started my PhD. when touch was a field that didn’t receive very much interest compared to other fields like computer vision or natural language processing. But to that time, when I started my PhD, we were considering enabling the surgeon to have a sense of touch for minimally invasive surgeries, and we were considering different sensors, and we didn’t find any suitable sensors in the market that can be deployed in minimally invasive surgeries. That’s how I studied my interest in the sense of touch. And back to that time, I discovered that it was very challenging to get the sense of touch in the very limited space, like minimally invasive surgeries.
01:46 · Shan Luo
And then I started working on designing tactile sensors and also these processing for tactile data. So that’s how I started my journey into the sense of touch and also tactile sensing.
02:04 · Michelle Sun
Amazing. And what were some of the biggest breakthrough in the field of tactile sensing in the recent five years, let’s say, and what are still unsolved in a field right now?
02:17 · Shan Luo
I would say in the past five years, the biggest breakthrough would be how we can scale the tactile data, especially in simulation and also in the real world data collection. We created this simulation for tactile sensors for the first time, and it was followed by many other researchers, including MIT and many others. So this enables the data collection in the simulation, and we can train our robot agents equipped with tactile sensing in the simulation so that we can transfer such robot agents to the real world.
03:01 · Shan Luo
And this has attracted a lot of interest from different fields. And we can see how this can be integrated with vision and other modalities and enable robot dexterity. And on the other hand, the development of models for processing tactile data is another breakthrough. So we have seen various models that can enable the understanding of tactile data and how to embed tactile data into the bigger ecosystem of the intelligent systems. I believe these are the two major breakthroughs for tactile sensing in the past five years.
03:40 · Michelle Sun
And what are still unsolved right now? What are the frontiers that you’re looking at that you hope that more people can work on it and it will lead to the next breakthrough?
03:53 · Shan Luo
Yeah, I take the interface as the missing piece of the whole building blocks. So we have seen different tactile sensors that we have seen these models. But we haven’t seen an interface to integrate different tactile sensors into this ecosystem. So different tactile sensors, they have different advantages and disadvantages. For example, like vision-based tactile sensing, they have high resolution for these different areas of the body parts. So vision-based tactile sensing can be challenging to be equipped to these body parts. For other tactile sensors like magnetic-based tactile sensors, they are very fast and they can be made very small.
04:46 · Shan Luo
But the resolution can be a challenge. So that’s why we need different types of tactile sensors in the ecosystem. And we need the interface so that the different tactile sensors can talk to each other. And these can be integrated into the whole intelligent ecosystem.
05:04 · Michelle Sun
For sure. One of the things that when I first learned about tactile data is that the sensors are also different and they produce different types of data. And so we’ll definitely talk more about this later in our conversation about your work to help integrating various data as well. And so let’s take a step back and talk about what touch is for when it comes to robotics and manipulation tasks. So let’s talk about the manipulation task that requires tactile sensing today. What are some tasks that touch is the requirements?
05:46 · Michelle Sun
Without a touch, it cannot happen. And then what are some tasks that touch is an improvement to the task?
05:55 · Shan Luo
Yeah. I would say for this task that require physical interactions, the touch will be a requirement, not improvement. So I will give three examples of such tasks. So one can be material assessments. For example, like hair assessment. This is a daily task for humans. So for vision, we can see the hair. But we can’t estimate how we can feel the hair. And really, when we get the contact with the hair, we can feel the hair. And we know the smoothness, stiffness, and other friction properties of the hair.
06:39 · Shan Luo
That’s the time when we use touch to evaluate the materials. So similarly, for other materials like garments or other materials in our daily life, we really need to use touch to evaluate their material properties and how they feel. So that is one example. And we have been working on this with Unilever. As Unilever, they have these products like shampoos or conditioners. And in the normal practice, they recruit people to assess hair or fabrics using their own hands so that they can evaluate how much friction when the hand is interacting with the hair.
07:25 · Shan Luo
But however, this can be biased and this can be very costly for their product development. So they have been working with us to develop these tactile sensing solutions so that we can use artificial touch to evaluate such materials. And this cannot be done by vision. And vision can only give you the appearance information. So that is one example. Another example is wiping a table. So how we clean a table using a wipe. So in that case, vision is definitely occluded by the hand. And we need to use a sense of touch to detect how much force we have applied to the table.
08:10 · Shan Luo
And if we detect any dirt on the table and how clean we have cleaned up the table. So this is another example. We need to rely on the sense of touch in doing the task, not the vision, as the vision is occluded by the hand itself. Another example is how we can insert a USB or a connector into our socket. So in that case, vision cannot feel how we have inserted the USB into the socket. So we need to use a sense of touch as the fingers to feel how much reaction force we can get from the socket.
08:53 · Shan Luo
So for the interactions between the socket and the USB. So only using the sense of touch, we can insert the USB into the socket. But that’s not with vision. So these are the examples. I can say we rely on tactile sensing instead of vision. So in that case, tactile sensing is not an improvement, but it’s a requirement.
09:18 · Michelle Sun
I love that. And I love that you’ve given examples of both from a commercial standpoint with Unilever and then from a home setting, where now a lot of people look at the home robots. And they’re hopeful to have these robots helping to do their chores. And then also on just a daily life standpoint like inserting a cable into a USB plug. And in terms of when you think about vision only manipulation, there’s been quite a lot of improvement over the years. How has that shaped your view of tactile sensing and where tactile sensing fits in the manipulation stack?
10:01 · Shan Luo
Yeah. I have seen many developments in vision-based manipulation. And this has improved the manipulation task a lot. But I would say in many of these tasks, vision can get the robots to do the task, but not complete the task. And this turns out to be clearer for robotics researchers that tactile sensing is important in completing the task. So usually when we have these objects on the table, we use vision to know where the objects are on that table. But when we pick up these objects by the robots, so when the contact is established, tactile sensing is much more important than vision.
10:55 · Shan Luo
As this tactile sensing can tell you how the contact dynamics happens between the fingers and the cup, as this may change the state of the objects. And the state of the objects cannot be perceived by vision only, as vision can be occluded by the object or by the hand. So in that case, tactile sensing is important to complete the task. And I would call it like last mile problem solver for manipulation task.
11:26 · Michelle Sun
For sure. That’s definitely a great way to frame it, right? So vision is great to start and probably do 80%, 90% of the task available. And then the last mile really to complete the task in all sorts of conditions where vision cannot even see or is occluded, then tactile really comes into play. And I think as humans, we take it for granted, right? Like the sense of touch gives us so much information. And sometimes we don’t think about it. And we think about like our eyes or our vision more.
12:01 · Michelle Sun
But actually, the touch has been informing us a lot, like feeling what’s in our pockets, finding out things, and without having to pick up the stuff in our pockets and show it to our eyes. And another thing that you worked on was about transparent objects. How do you see that? Do you think that’s one of the best proofs about touch being in a requirement? I know you shared like three other really great examples as well. Is there like a strongest argument for touch is the requirement for manipulation?
12:40 · Shan Luo
Yeah, I would like to put it in this way. Touch is a requirement for completing the task, especially in this example of transparent objects manipulation. So in that work, we have done some experiments to compare vision-only manipulation and also vision-guided touch manipulation. So in these two scenarios, if we look at the depth maps from vision of these transparent objects, so it’s quite sparse. We cannot detect these objects on the table. And it’s very challenging to use vision-only information for the manipulation. However, in our model pipeline, so we use vision to guide touch to pick up the objects.
13:34 · Shan Luo
In that case, vision can give us some sparse cues of the objects. So for example, the edges of the cup, especially the top edges of the cup, even though the body of the transparent object is very challenging to detect by vision. But it can give us very sparse information so that we can use touch to verify if the edges are there and we can estimate how the object is standing on the table. And then touch can complete the task of picking up the objects.
14:14 · Shan Luo
So with this example, I would say vision can give us some global view, can give us some initial estimates of the object state. But touch will be the one that will complete the task in the whole loop.
14:29 · Michelle Sun
That’s a good segue into GenForce. You publish GenForce, which is when you take a step back. So GenForce is the research you’ve done where you took a force model trained on one sensor and make it work on another model and on another sensor. So as we mentioned earlier, the tactile sensors are such a different challenge, different from cameras or other types of sensors, where the data that it generates are very different, sensor to sensor types. For someone outside of the field that are not familiar with tactile sensing, what does GenForce make possible?
15:16 · Shan Luo
I would put it in this way. So with GenForce, we significantly reduce effort from a sensor-specific calibration to a transfer problem. So traditionally, we have these sensor manufacturers, and we need to get these calibration models for each sensor. And even for the sensors from the same manufacturer or the same sensor, if we use it for a certain period of time, so the force prediction will be very different from the force predictions out of the factory. So it means we need a model, and we need a lot of data to calibrate the sensor so that it can meet the requirements for a certain task.
16:07 · Shan Luo
But with GenForce, we can have the force prediction model trained on one sensor to be adapted to another sensor. So for example, if we have a model for force prediction trained for GelSight, the vision-based tactile sensor, so this model can be adapted for another type of sensor. It can be a vision-based tactile sensor like TacTip, or a GelSight sensor with a different elastomer, or a tactile sensor of a totally different sensing transduction sensors, for example, like a uSkin from XELA with magnetic-based tactile sensors.
16:50 · Shan Luo
So in that case, we don’t need to collect a large amount of data to return the force prediction models for the new sensor. Instead, we just use the prior knowledge trained for another sensor, and then we can adapt it for another sensor. So in that case, we can save a lot of efforts for the force prediction for different sensors. And also, in the same time, as we use the sensor, we can dynamically adapt our force prediction models for the same sensor so that it can keep the accurate predictions of the force for the same sensor.
17:29 · Michelle Sun
This kind of reminds me of vision, where vision data compound, because different data from different cameras generate the same type of data. Different cameras generate the same type of data, and that can be used by the same vision model. With GenForce, you’re kind of unlocking that capability, where tactile data set become the assets, and the sensor becomes the inputs. In this scenario, with GenForce becoming more powerful, does the sensor hardware commoditize? And if it does, who ends up holding the cross-sensor data set?
18:11 · Michelle Sun
Is that a company or a consortium?
18:13 · Shan Luo
So there are two questions here. So first, does the sensor hardware become commoditized? So to some extent, yes. But I don’t think the hardware becomes unimportant. So what GenForce start to show is that the data and knowledge learned from that data don’t necessarily have to remain locked to one particular sensor. So if I collect a very valuable tactile data set using one sensor, and I can transfer what I have learned to another sensor, so then the values are to shift from the individual piece of hardware towards the data models and also representations build on top of it.
19:01 · Shan Luo
But hardware still matters a lot. So different sensors have different spatial resolution, sensitivity, bandwidth, durability, and also form factors. As I just mentioned, so for different body parts, so we may need different types of tactile sensors. So that’s why the tactile sensor hardware is still very important. So you can’t recover information that the hardware never measured in the first place. So I see this less as hardware disappearing and more as hardware becoming part of a larger interoperable tactile ecosystem. For the second question, so who should host the cross-sensor data set?
19:47 · Shan Luo
My preference would be for something closer to a consortium or standardization organization rather than a single company owning everything. So for tactile sensing to reach something like the scale that computer vision achieves, we need many groups to contribute data from different sensors, robots, objects, and manipulation tasks. But for those data sets to become generally useful together, we also need standards. So apart from the sensor specification, resolution, durability, and many other metrics, we also need these standards to standardize the data collection process to benchmark different sensors.
20:33 · Shan Luo
For example, what object stimuli we should use for benchmarking such sensors, like how much force, which trajectories we need to use to contact the tactile sensors. So these are the protocols we use for data collection. So only with that, we can standardize our data collection, our datasets, and we can benchmark these models for tactile sensing. So in essence, I believe the best practice would be to have the consortium or standardization organization to own these datasets or these protocols.
21:14 · Michelle Sun
For sure. And you touch on a really important point where different parts of the robot body would require different types of tactile sensing. And so that’s why where the tactile sensing hardware actually plays a really important part to contribute to this cross-sensor data sets. And you worked on GenForce at your lab, which makes tactile data portable across hardware. And you also worked on ViTacGen, which is generating tactile signal from a camera, and it is up to 86% success rate using vision alone.
21:57 · Michelle Sun
So on one hand, GenForce is saying that data is the assets and it’s portable across different hardware. And for ViTacGen is saying that the signal is able to transfer without the hardware, the tactile side of the hardware. Which of the two changes more for a company that sells tactile sensors today?
22:21 · Shan Luo
Yeah, I actually think both are right, and they are complementary rather than conflicting with each other. So GenForce asks a question, if I need physical touch, can the knowledge I learn from one tactile sensor transfer to another? And the answer is yes. So that makes tactile hardware easier to deploy because new tactile sensors don’t require us to start again from zero. But on the other hand, ViTacGen asks a different question. So once I have learned from this physical touch hardware, do I always need the tactile sensor?
23:03 · Shan Luo
at deployment time. And in some tasks, the answer may be no. And in the ViTacGen work, the task we were addressing is object pushing. So in that task, we just use an end effector of the robot to push one object from position A to position B. So in that case, the real tactile information we need is the geometry information of the objects. So in that sense, we can transfer the geometrical information learned from tactile sensing to vision from its views. And so in that sense, we don’t need the tactile sensing in the deployment once we had trained the models using tactile sensing and vision together.
23:50 · Shan Luo
So with ViTacGen, we use tactile data during the learning process to teach the system about physical interaction within the relationship between what the camera sees and what the tactile sensor would have perceived. So that’s why at deployment for some tasks, the knowledge can be carried by vision without requiring the tactile hardware to be present. But I don’t interpret ViTacGen as saying that the tactile sensors are not needed. So in fact, we need tactile sensing to create the physical experience that the model learns from in the first place.
24:30 · Shan Luo
So we can learn from humans as well. So we physically touch objects and materials and build associations between how things look and how they feel. So later, we can look at something and predict that it will be soft, rough, or slippery without touching it. But that prediction comes from experience with the physical world. So for tactile sensor company, I think these results point towards a broader business than simply selling a piece of hardware. So the sensor generates the physical experience. So the data captures that experience, and the model makes that experience transferable.
25:15 · Shan Luo
And then, so I think about the whole stack, so we need the hardware to gather the initial data. But also, the tactile data can tell us the experience, how we can interpret the physical world, and also the models that can turn that data into reusable physical intelligence. So that’s my standpoint for these two questions.
25:42 · Michelle Sun
Yeah, that’s really cool. And I see that the tactile sensor data actually is also the input for ViTacGen, in addition to human interaction data, to make that prediction better. So it’s actually important in the whole stack and important in improving the whole loop. And your lab currently has all these different sensors, so GelSight, TacTip, uSkin hardware, et cetera, side by side. What do you replace the most often? And how often do you have to replace them?
26:19 · Shan Luo
That’s actually quite a different question to answer. So it really depends on what experiments we have. So we have various tactile sensors, like the GelSight Mini, we have TacTip, and we have uSkin. Most of them were donated by this company to our research lab. And we also make our own sensors to fit our task requirements. So from our experience, we may replace tactile sensor after a certain period of time, as the extensive use of the tactile sensor will make the elastomer of the tactile sensor [unclear].
27:18 · Shan Luo
Especially for various tactile sensors, we have this painted area, we have the painted layer. So this may get thinner as we have extensive use. So usually, we may say after a few months of use, so we may replace these elastomers. But for some other sensors like uSkin, as we have a rubber material, it can be more robust. And we may replace such material for a longer time. But this is similar to our human skin. So we get new skin after a certain time. So we are not having the same skin as the one we were born.
28:04 · Shan Luo
So that’s why it’s similar for artificial tactile sensors, as this skin is the part that gets most physical interactions with the environment. So that’s why we may need to replace such elastomers for a certain time. And also, it depends on what task we have. So if we have gentle contact with objects, we may use it for a longer time, like one year. But if we have extensive shear force, so this can make this elastomer easier to get damaged. So we may replace it in a shorter time.
28:43 · Michelle Sun
Yeah, so around anywhere between a couple of months to a year. And you touch upon the elastomer, so I wanted to follow up on that. That’s an interesting parallel that we’ve seen with the human skin. Our skin gets shedded throughout the year. And essentially, it’s always changing. It’s almost like a replaceable layer for our body to shield against the environments. Or let’s say this robot starts to also have consumable skin. Do you think the replacement cycle would be sometime around a year, a couple of months?
29:23 · Michelle Sun
And how much would it cost per robot per year to have a skin like that?
29:32 · Shan Luo
I would say it really depends on what tactile sensors you may have. As the skin we use for vision-based tactile sensor could be very different from the ones we have for magnetic-based tactile sensors. But normally, we take such skins as consumables. In practice, this may cost a few dollars for the skin, a small patch for the finger pads. So in that sense, if we replace the skin after like three to six months, so the cost is not high. But the real question is, can we make sure the model can still work if we change the skin?
30:20 · Shan Luo
So that’s a real tricky question for now. If we change the elastomer, the material property may be slightly different. And also, the alignment of the elastomer with the sensor can be slightly different. How can we make sure the models we have developed for the previous skin can be reused for the new skin? And that’s how we can use GenForce to make it happen.
30:50 · Michelle Sun
And your outlook paper also said something about how manual the process right now is when it comes to making these sensors. So optical sensors are made by mixing, painting, coating, all done by hand. And e-skin needs many manual steps in the clean room. So for touch to... and you mentioned that from different skins, sometimes the model doesn’t work, and maybe in simulation, it needs to be tweaked as well. So for touch to ship on a production robot, what matters more? Is that taking the price down further or getting every unit to read the same so that the model works across different hardware?
31:39 · Shan Luo
I want to say both. So on one hand, if we push the tactile sensors into the market, so we need to get the price down so that it can scale up. It can be used by the robots. That it will be deployed in factories, in homes. So that’s why we need to minimize the cost of tactile sensors so that we can minimize the total cost of the whole robot. So that is one hand, and definitely it’s the trend to minimize the cost. On the other hand, so we really need to make sure how we can scale up a different tactile sensor.
32:16 · Shan Luo
As we just mentioned, we may need different types of tactile sensors for different parts of the hands or different body parts. So in that case, how we can have different tactile sensors talk to each other, how we can have this interface between different tactile sensors, that’s OK. How we can get tactile sensing into the bigger intelligence system.
32:38 · Michelle Sun
For sure. So let’s shift gears and talk about TacDiff a little bit. Your view is that robot parts get built first, and algorithm shows up afterwards. And with TacDiff, you’ve been spending five years designing the sensor shape and the control policy together. Where is the biggest gain when it comes to getting these capabilities up? So is it a better sensor or designing the better policy and the finger at the same time?
33:12 · Shan Luo
Yeah, I think the bigger gain comes from designing them together, because there isn’t really a universally best tactile sensor or robot finger. And this is a better design for a particular task. And this is based on our experience in designing the tactile intelligence systems. So for example, using a parallel gripper, using a cube-like tactile sensors. So we can pick up objects like the phones or cups. But if we use such a parallel gripper to pick up a piece of fabric, a piece of paper, it’s very challenging. So the main reason is that the form factor cannot enable the picking up.
34:00 · Shan Luo
If it is a cube-like form factor, it’s very challenging to adapt such a sensor to pick up a soft object of very thin objects. So that’s how we design these RoTipBot, with a finger like these cylinder shapes, like our human finger, so that we can roll over the objects. And we can use the shear force to pick up the object, like a piece of paper or a piece of fabric. That is not possible by a cube-like tactile sensor for a parallel gripper. So this gives us the idea how we can design form factor and also other parameters, like control policies and also these material properties of the elastomer altogether to optimize its designs for a given task.
34:55 · Shan Luo
So all of these will be dependent on the task given by the user. So that’s why we propose TacDiff. We want to use simulation to optimize these different parameters, so that we can optimize these parameters for a given task. So with TacDiff, we want to reverse the traditional thinking. So we go to one design, so traditionally, we don’t consider what task we want to have. We don’t want to guess the best design. It results in the considerations of specific tasks. So here, we start with the task and ask, what morphology, what sensing and what control policy should the robot have to perform this task well?
35:46 · Shan Luo
So take manipulating a very delicate object as an example. So you may want a soft fingertip with high force sensitivity so that the robot can detect very small contact forces. But if the task is inserting a component during assembly, the optimal finger might be stiffer with sensing focused in particular areas where contact tell you about alignment. So this example can give us more motivations to use these simulations that can optimize these design parameters. So ultimately, the question changes from what is the best tactile sensor to what is the best tactile robot for this task.
36:34 · Shan Luo
So here, we use TacDiff, this differentiable simulation so that we can optimize these parameters based on the given task. And then we can get better performance for that task.
36:47 · Michelle Sun
For sure. And in the beginning of the podcast, you mentioned about Unilever, the work that you’ve been doing on the shampoo and testing the texture of the hair. And you also work on the deformable and packaged goods as well. With an industrial partner like this, what have you learned from working with a partner like Unilever? And what are some things that required of your work that your research have not been producing? And you have to work with them to solve some of the problems.
37:23 · Shan Luo
Yeah. So there are some problems that we would like to solve together with our industrial partners. Taking Unilever as an example, so in their practice, they recruit people to assess hair so that they can have subjective assessment of hair, and then they can evaluate their hair properties. Also, they have some objective measurements practice. They use this tribology technology to collect the friction data, and then they have this simple linear regression model to predict these hair properties. But the information is very limited, collected from the friction data.
38:15 · Shan Luo
So compared to their practice, using these tactile sensors we have developed, these vision-based tactile sensors, we can capture much richer information. For example, the fiber distributions, the texture of the hair, that cannot be disclosed by the friction data collected from the load cells. So this can give richer information that can be comparable to subjective assessments. So that’s what we have solved, and we can map this richer tactile data into the space that can be aligned with subjective assessments. That was not possible before our solutions. Another example is with Ocado technology.
39:09 · Shan Luo
So Ocado is the largest online grocery in the UK, and they have been operating its automated warehouse. But in that case, they still rely on human resources to handle these delicate objects, like fruits, these delicate objects. And they cannot detect these forces between the interactions of the objects with these robots. So with tactile sensing, we can detect these forces with delicate objects, and we can moderate the forces applied to these objects. So these are the problems that we are solving in the industry, and they are not possible without tactile sensing.
40:01 · Michelle Sun
For sure. And you’ve founded the ViTac workshop, and this year is the seventh year at ICRA. The theme was learning to see and feel. Every year, you see where the whole field of tactile sensing has been developing, and what are they submitting to the workshop. What has stood out to you for this year, and what are you seeing that are emerging as trends that you’re excited about?
40:32 · Shan Luo
Yeah. The biggest change I have seen over the past two years is transition from hardware-focused research to intelligence-focused research. So when I started the ViTac workshop series, so many of the papers we received were on the hardware, how we designed the tactile sensors of different sensing principles, different form factors that we can equip such tactile sensors to different types of robots. But recently, we have seen many papers working on foundational models, simulations for tactile sensing, how to scale up data collection for tactile sensing. So we are seeing more intelligence elements in the submitted papers.
41:24 · Shan Luo
Definitely this is a trend in the community. So from what I can say, it’s also a reflection of what we have developed over the years. So we can see intelligence can be built on top of its hardware sensors. So we have seen a convergence of the sensor developments. So we have got good sensors for these robots. But now we want to answer the question, how we can use such sensors in robotic systems? How we can make it benefit these tasks? So the questions have changed from, can the robot feel?
42:06 · Shan Luo
To what can the robot learn from feeling? So that’s the transition I have seen in the past years.
42:16 · Michelle Sun
Yeah, so from hardware-driven design to intelligent focus research. And that actually ties everything together really well, with the more and more tactile data being collected in the real world in deployment, not just in the labs. And how do we make use of it and to build better model for the robots? And you hold a Turing open source AI fellowship. In tactile, which layer benefits most from being open? Is that the data sets, the trained model, or the sensor designs?
42:56 · Shan Luo
Ideally, all three, so the data sets, the models, and the sensor designs. So they benefit from being open. So the community can benefit from open sourcing these designs. So this can accelerate the designs of tactile robots. But if I had to prioritize, I would put data sets and models first. So the reason is that their value compounds, so tactile data, are still relatively scarce compared to with vision or language. So every research group tends to have their own sensor. So its own experimental setup and its own data sets.
43:42 · Shan Luo
So if those data sets remain siloed, every group has to keep collecting essentially the same kinds of physical interaction again and again. So that’s why I would say the tactile data collection is the first priority I would like to highlight to be open-sourced so that we keep the same or agreed protocols for data collection. And we can share such data sets across research groups. So then we can interoperate these models across these data sets. So the models can then be deployed across different data sets. And so these models can be benchmarked.
44:33 · Shan Luo
So we know how the model can progress. And this will be similar to what we have already got for ImageNet. So we have this data set that is benchmarked, well documented. And also the models can be benchmarked. So similarly for tactile sensing, we need the data set first. And then we need to benchmark such models. And we can reproduce these results across different sensors and across different tasks. So that’s my view for open-source studies.
45:11 · Michelle Sun
Yeah, it’s interesting that it actually is a sequential thing, right? So the data set actually unlocks better models. More data unlocks better models. And then it informs better designs for the sensors. So I know that, Shan, you’ve been in academia. And I’m sure a lot of people ask you this. But if you were starting a company today instead of running a lab, would you build a sensor, a data set, or a simulator, or a different answer?
45:43 · Shan Luo
Actually, I have already started the company. It’s called Hand Intelligence. So we are building this venture. And to answer your question, I wouldn’t choose just one of these three. So what we are building is really a full-stack physical AI company. Because I think the sensor, the data, and the models are deeply connected to each other. So you need a sensor to interact with the physical world and collect the high-quality tactile information that generates the data, which I think will become an increasingly important asset, as we discussed earlier.
46:24 · Shan Luo
And then you need the models and simulation to learn from the data, scale it, and transfer that physical intelligence across different sensors, models, and tasks. But if you only have the sensor or the data sets, you will be limited to being a hardware supplier, or you don’t have a way to generalize to different tasks. So that’s why we also developed these AI models, including these foundation models and world models that is embedded with vision and tactile sensing, so that we can complete these physical tasks.
47:06 · Shan Luo
So I think the long-term value is not simply in giving a robot a better fingertip, it’s in giving robots a better understanding of physical interaction. So as a very first step, what we are developing now is funded by ARIA. So ARIA is a funding body in the UK, and they have a Robot Dexterity programme through this program. So Hand Intelligence is developing this interface so that different tactile sensors can talk to each other. And we are generating these unified representations that can have different tactile sensor manufacturers can adopt these unified representations, and we can get these benchmarks or representations for different tactile sensors.
47:55 · Shan Luo
And they can be used for downstream tasks of robots. So if you ask me, sensor, dataset or simulator, my answer will be the interface between all three is where I would build the company. And that’s actually what we are doing at Hand Intelligence.
48:16 · Michelle Sun
How do you spell it? Like just for our notes, like Hand Intelligence, is it like hand?
48:22 · Shan Luo
Yeah, Hand Intelligence.
48:25 · Michelle Sun
I see, I see. Is that live right now? Like just for our editing, do you want this to be mentioned in the podcast or like when are you guys going live?
48:37 · Shan Luo
Yeah, we are still at a very early stage of the venture. And so we want this company to be mentioned as this can be heard by like investors or any interested companies. So this can be mentioned to the audience, the podcast. Also, at the same time, we are still as the stealth mode. So we are not disclosing very much detail of the company. But we want people to know we are doing this already.
49:16 · Michelle Sun
Awesome, awesome. And if you have a link or any website that I can point to, then we can include that into the show notes and the editing as well.
49:28 · Shan Luo
Yeah, at this moment, we don’t have a dedicated website yet. Yeah, so if any of the audience is interested, so they can reach me. And I’m happy to talk more about our company.
49:43 · Michelle Sun
OK, sounds good. Let’s wrap up the last question. And then I’ll stop the recording. And I’d love to learn more as well. So this is exciting. And I’m sure with the company, you guys thought about it a lot when it comes to the next five years of where the industry is going. So take me to 2031, a robot works eight hour shift in a warehouse. How much of that shift involves hand intelligence, touching something, and what is on its fingers?
50:14 · Shan Luo
Yeah, in five years, I would imagine that these robots have been deployed in warehouses. And each of the robots has tactile sensors equipped to their hands. And they can detect the contacts, the force interacting with the objects. And they can decide how much force can be applied to these objects, and it can better complete this task. So we have seen some demos. They are just looking at these boxes. And they are not really feeling the contact with these objects for now. But in the five years, I would say when these tactile sensors are in their hands, they can really tell which objects they are touching, what material they are touching, and how much force they need to apply to that object.
51:08 · Shan Luo
And this will close the loop and help them to complete the task.
51:14 · Michelle Sun
Amazing. Well, thank you so much, Shan, for your time. It’s great having you today.
51:18 · Shan Luo
Thank you, Michelle, for the invitation. And it’s great to share what we have got so far with the audience.