Why I built an open source robotics dev framework
After closing the chapter on selling my second company, MonkeyLearn, I started thinking about what came next. I knew I wasn't made to just stop working. My brain needs something to work on. It's what I enjoy.
This is the story of what came next: Solarian, an open-source Python framework for building AI-powered robots, and a home for the open hardware and models that go with it. It's also the story of why I think a robotics framework still makes sense in 2026.
This time, I had the time and resources to think deeply about what I really wanted to do. I wanted this to be more than another business. Something closer to a life's work.
At the same time, software, the industry I know and love, was changing dramatically. I've been questioning how much of the SaaS industry will still make sense when software becomes so cheap and easy to build with AI agents. My next project needed to be something I deeply loved, but also something that could build strong moats in this new environment.
Enter robotics.
Robotics was already at the top of my list. I've been playing with it since 2008, when I built my first sumo robot. The thing holding me back was the well-known difficulty of building a hardware company. Meanwhile, most of my software ideas kept falling off the list: I either wasn't passionate enough about them or couldn't see a strong enough moat.
So I decided to make peace with the fact that a robotics business could be extremely hard and start building something I truly loved. Thanks to the exit, I had the privilege of choosing what to work on. Why not use it? I gave myself permission to just start playing with robotics, without the pressure of building a business around it.
I started experimenting with robots and reading a ton of research papers on the state of the art in AI for robotics. I met people in Silicon Valley, went to robotics meetups, conferences, and talks at Stanford. I even started a graduate program in robotics there. I went to hackathons, saw what people were building, explored their tools, and learned about the difficulties they faced.
I started playing around with ROS, then used it more formally at Stanford. I also started using LeRobot, the robotics library from Hugging Face. They were very different tools, and my experience with both helped shape what I wanted to build.
I used ROS 2, the current generation. Despite its name, Robot Operating System, ROS is a collection of libraries and tools for building robotics applications, with a large, established developer community. But when I started using it, the experience wasn't what I had hoped for.
I didn't find the documentation clear or inspiring. It felt far from the developer experience I was used to in the Python and JavaScript communities. Setup was frustrating, particularly on a Mac, where my workflow involved running Linux in a virtual machine. To me, that was a lot of friction before I could even start building.
The architecture didn't feel intuitive either. Organizing a system around nodes communicating through topics felt too low-level. Debugging behavior spread across those nodes was even more frustrating. It also felt like a framework designed for an earlier era of robotics, centered on explicitly programmed control logic. I wanted something built around the newer world of learned control: ACT, diffusion policies, and vision-language-action models (VLAs).
In conversations with people in academia and industry, I kept hearing similar frustrations. Some companies had moved away from ROS and built their own systems. Those conversations reinforced my feeling that there was room for a different approach.
LeRobot, on the other hand, was built around making modern AI for robotics accessible to developers. It was gaining a lot of popularity among robotics enthusiasts, and I could see why. The team did a great job bringing together Python implementations of methods like ACT and diffusion policies, later adding VLAs, along with support for accessible hardware using Feetech and Dynamixel actuators.
But I think one of their most important contributions went beyond the software: helping people get their hands on affordable, open-source robots. The ecosystem included arms like the Koch, SO-100, and SO-101. The SO arms, designed by The Robot Studio in collaboration with Hugging Face, made it possible to get started with arms you assemble yourself, at a few hundred dollars per arm. Closed-source robot arms often cost several thousand.
That was huge. Hardware is a major barrier to entering robotics, both because of the cost and because building it can be difficult. Making it accessible to an individual developer opened the door to a much larger community.
I dove deep into LeRobot. I found bugs I wanted to fix and features I wanted to add, and I was excited about contributing. But the deeper I went into the code, the more I realized that the architecture wasn't the foundation I wanted for what I had in mind. The changes I wanted to make went beyond adding features. From my point of view, they would require a substantial rewrite.
One of my biggest frustrations was how hard it was to build reliable robots with behaviors I could reuse. In the workflows I was using, each project seemed to start over: collect demonstrations for a particular robot, train a policy for that embodiment, and start again from scratch if the hardware changed. What I was missing was a standard interface that made those learned behaviors portable and reusable across robots.
It wasn't just the arm. Everyone seemed to be using a slightly different gripper, and camera setup. Different camera models, different numbers of cameras, and different camera positions, sometimes changing even between recording sessions. We were changing both how the robot moved and what the model saw. In the setups I was working with, I couldn't simply take a trained model and expect it to work somewhere else, zero-shot.
Fixed third-person cameras were another problem. Mounting a camera outside the robot ties the setup to a particular workspace. That wasn't the direction I wanted for a mobile manipulator: an arm on a wheeled base, a biped, or a humanoid that moves from place to place. I wanted the cameras to travel with the robot, with a consistent setup across data collection and deployment: matching camera intrinsics, such as lens geometry and field of view, and fixed extrinsics relative to the gripper, meaning the camera's position and orientation on it.
The VLAs I was exploring offered a way to start with a pretrained model and fine-tune it for a particular robot. That was valuable, but it didn't give me the kind of reuse I wanted. I wanted to take an existing policy and run it on another arm, even one with different kinematics, without collecting more demonstrations and retraining it.
That got me very interested in UMI, the Universal Manipulation Interface, from researchers at Stanford, Columbia, and Toyota Research Institute. To me, it's one of the most influential pieces of recent robotics research.
UMI lets you collect demonstrations with a handheld gripper and learn policies that describe the gripper's motion rather than a particular arm's joint movements. The same policy can then run on different compatible arms without additional training. The researchers demonstrated this with a UR5e and a Franka.
Its camera setup was a big part of the appeal: one camera mounted on each gripper, with no external third-person camera required. A fisheye lens gives it a wide view of both the manipulation itself and the surrounding scene. That keeps the visual interface consistent when you move it to another arm or to another place, while giving the policy much more context than a close-up of the fingers. It also got me thinking about how that wider view could be useful for navigation once I moved toward mobile manipulation.
The arm needs enough degrees of freedom and reach to execute the motion, with inverse kinematics translating it into joint commands. The camera and gripper setup also need to match the policy's interface, and observation and action timing matter. But within those requirements, changing the arm doesn't have to mean retraining the policy.
That was much closer to what I had in mind.
The other thing I found incredibly valuable was how you collect the data. With UMI, you demonstrate the task directly using a handheld device: a 3D-printed gripper with a GoPro recording video and IMU data. You don't need to teleoperate a robot through a leader arm. You just pick up the gripper and perform the task.
Collecting a demonstration with the handheld UMI gripper. Video: the UMI team at Stanford University, with Columbia University and Toyota Research Institute.
That's huge for several reasons. A handheld collector can cost much less than a complete leader-and-follower setup. It's also portable. Most of the leader-arm setups I was seeing were bolted to a table. With UMI, you can take the device into a kitchen, a workshop, or even outdoors, wherever the task actually happens, and collect demonstrations there without bringing a robot along. And I found it much easier and faster. Teleoperating through a leader arm takes practice. Moving a gripper directly with your own hand feels much more natural. That matters because scaling robot learning means collecting a lot of real-world data. The cost, effort, and time needed for each demonstration make a huge difference.
Getting the original UMI code running reliably was another story. At the time, I wasn't finding the ongoing maintenance and documentation I needed. It took me months to get it working: fixing small issues, trial and error with the robot, and figuring out details that weren't documented or fully explained.
When it finally worked, my robot ran UMI's cup arrangement policy successfully for the first time:
Once UMI was running, I hit another limitation. As far as I could tell, most policies, UMI included, gave you no way to tell the robot which object to interact with. If a policy was trained to pick up an object, you couldn't choose which one it should pick at inference time. VLAs let you name the object in a text prompt, but text is a clumsy way to single out one object, especially among similar ones, and VLAs often struggle to follow exact instructions.
I felt the prompt should be visual instead, like a bounding box or a segmentation mask around the target object. That idea led me to some original work, which became my final project, with my teammates, for Stanford's deep reinforcement learning course, CS224R: Point and Pick. Now I could point to the target object and have the policy act on it. For me, that was a major breakthrough. (You can browse all the CS224R final projects too.)
So after all of those experiences, from ROS and LeRobot to UMI and Point and Pick, I did what an opinionated engineer would do: I decided to build my own framework.
I wanted the accessibility of a modern Python library, an architecture that felt intuitive to me, and AI-based control as a starting point. I had enough knowledge to get started, and I had the time and resources to keep going.
That's what I've been working on, somewhat quietly, for the past year and a half: Solarian.
I adopted UMI as the main approach to learning manipulation policies in Solarian. I think making that approach easier to use and reuse is one of the most important things we add: taking a powerful research idea and doing the engineering work needed to make it practical for other developers.
The project began at a very particular moment in the history of software development. I started writing Solarian back in May 2025, when typing code yourself was still a much bigger part of software development. My workflow was mostly an IDE with very intelligent code completion. I started with Copilot, then moved to Windsurf, and finally Cursor.
I wrote the foundations of Solarian by hand. As the great DHH would say, I was chiseling the code myself, making decisions about how each piece should fit and how writing a robotics application should feel.
I think starting that way helped me stay close to what I really wanted to build. It gave me room to put my own taste and opinions into the foundation.
By the time I'm launching Solarian, in November 2026, AI coding tools have moved toward agentic engineering: you describe what you want, and agents go off and implement it. Many developers are now comfortable doing this without reading much of the generated code.
I definitely ask AI agents to build things for Solarian. But the core architecture was already established, and I still work mostly in "IDE mode." I build one feature at a time, read the code diffs, and review each addition before accepting it.
Still, I've been asking myself the same question almost every day: does it make sense to build a robotics framework in 2026?
When I started in 2025, it already felt like the world was changing fast. It has changed even faster since then. But my answer is still yes. Here's why.
First, AI agents need building blocks too. They use open-source libraries, just as human engineers do. Reimplementing everything from scratch takes time and tokens, and it means solving problems that someone else has already worked through. I don't think we yet have the robotics framework I'd want an AI agent to reach for. That's the role I want Solarian to play.
I've spent a year and a half working through these problems, with plenty of AI tokens along the way: trying things, finding out what didn't work, changing the design, and discovering requirements I hadn't thought of at the beginning.
And I've tested it on real robots. That part matters a lot. Physical testing still requires hardware, hands, and someone dealing with what actually happens when a robot moves or breaks. It's much harder for an AI agent to do that on its own than to generate another piece of code. Software that moves hardware in real time faces a different set of challenges, because physical products don't live in a perfect world the way pure software does. Things break, behaviors are much less predictable, and hardware is not 100% reliable. The physical world is messy.
Again, it doesn't make sense for a human engineer or an AI agent to repeat all of that work from scratch.
Second, a framework is an opinionated way to solve a problem. It defines an architecture, conventions, contracts, and standards. It makes decisions about what should be easy, what should be explicit, what the priorities are, and which trade-offs are worth making.
I believe that is still harder for an AI agent to develop on its own. It comes from experience, frustration, feelings, preferences, intuition, and a vision of how things should work and feel. Why do it this way? What matters most? How should it feel to build with it?
Those are the decisions I care about, and they're embedded in Solarian.
Third, I still think the human developer experience matters. Some qualities of a framework may become less important as agents write more of the code: how quickly a person can learn it, how easy it is to read, how convenient it is to extend, how expensive it is to maintain. But humans still need to understand the systems they're building. We need to be able to read the code, question the architecture, and exercise judgment. And I suspect that clear conventions, good documentation, and a coherent architecture help AI agents for many of the same reasons they help us.
Finally, Solarian is more than a software framework. It's also a home for open-source hardware and models. The code is one part of what I want to build. Working with UMI eventually led me to redesign parts of the hardware myself. The GoPro was expensive, heavy, and difficult to extend in the ways I wanted. It also meant depending on one particular camera vendor.
So I designed my own camera device using off-the-shelf components: a Raspberry Pi Zero, a Raspberry Pi HQ Camera, an IMU, a display with joystick controls, and a battery. That gives me a portable computer with a camera and an IMU. I can change the software, connect it to Wi-Fi, add new sensors, and build new capabilities on top of it. It's cheaper, and I have much more control over what it can do.
I also designed an open-source actuated gripper for the robot itself. The original UMI setup used a Weiss Robotics WSG 50, which costs roughly $5,000. My design costs less than $100 to build. Those are different classes of hardware, and I wanted a much more affordable way for developers to get started with UMI.
And lastly, I designed a modification to the SO-100 robot arm that adds an extra degree of freedom, turning a five-DOF arm into a six-DOF arm. That gives it the additional freedom needed to control the gripper's position and orientation for UMI-style manipulation, within the arm's workspace and joint limits.
These designs are part of the same goal: make it easier and cheaper to collect demonstrations, build a compatible robot, and reuse what you've trained.
So what does that look like in practice? Solarian brings together the tools to build, simulate, teleoperate, train, and deploy robots in one Python framework, with the developer experience a modern framework should have. Here's what it feels like to define a robot and run a skill:
A few of the pieces I care most about:
UMI-based learning. The UMI plugin brings the approach I've described into Solarian, making reusable manipulation policies a central part of the framework.
Modular robots and reusable plugins. Build a robot from arms, grippers, cameras, and other modules, each running at its own rate. Install hardware support and skills as plugins, or write your own. Built-in inverse kinematics lets you command the end effector instead of individual joints.
Simulation alongside real hardware. Use the same robot definition and configuration in MuJoCo or on the physical robot, so simulation is part of the development workflow from the beginning.
A complete learning workflow. Record demonstrations, organize and transform datasets, train policies, and serve them back to the robot. Those steps belong in the same framework, with shared conventions for observations and actions.
An AI controller that uses skills. A vision-language model interprets the task and chooses which skills to run, while those skills handle the faster control underneath. A skill can be a scripted routine, a motion plan, or a learned policy.
The everyday development tools. Typed observations and actions, composable configuration, joint calibration, and latency measurement. These are the pieces I kept needing while building and testing robots, so I wanted them built in.
You can explore the framework, documentation, and plugins at solarian.dev.
I can keep debating whether building a robotics framework in 2026 makes sense. I probably will. But beyond all of those arguments, Solarian feels like a life's work.
To me, it's a piece of art. A gift from me to you.