#llm3

[sunaleisocial]

Teaching a robot its limits, to complete open-ended tasks safely

New Post has been published on https://sunalei.org/news/teaching-a-robot-its-limits-to-complete-open-ended-tasks-safely/

Teaching a robot its limits, to complete open-ended tasks safely

If someone advises you to “know your limits,” they’re likely suggesting you do things like exercise in moderation. To a robot, though, the motto represents learning constraints, or limitations of a specific task within the machine’s environment, to do chores safely and correctly.

For instance, imagine asking a robot to clean your kitchen when it doesn’t understand the physics of its surroundings. How can the machine generate a practical multistep plan to ensure the room is spotless? Large language models (LLMs) can get them close, but if the model is only trained on text, it’s likely to miss out on key specifics about the robot’s physical constraints, like how far it can reach or whether there are nearby obstacles to avoid. Stick to LLMs alone, and you’re likely to end up cleaning pasta stains out of your floorboards.

To guide robots in executing these open-ended tasks, researchers at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) used vision models to see what’s near the machine and model its constraints. The team’s strategy involves an LLM sketching up a plan that’s checked in a simulator to ensure it’s safe and realistic. If that sequence of actions is infeasible, the language model will generate a new plan, until it arrives at one that the robot can execute.

This trial-and-error method, which the researchers call “Planning for Robots via Code for Continuous Constraint Satisfaction” (PRoC3S), tests long-horizon plans to ensure they satisfy all constraints, and enables a robot to perform such diverse tasks as writing individual letters, drawing a star, and sorting and placing blocks in different positions. In the future, PRoC3S could help robots complete more intricate chores in dynamic environments like houses, where they may be prompted to do a general chore composed of many steps (like “make me breakfast”).

“LLMs and classical robotics systems like task and motion planners can’t execute these kinds of tasks on their own, but together, their synergy makes open-ended problem-solving possible,” says PhD student Nishanth Kumar SM ’24, co-lead author of a new paper about PRoC3S. “We’re creating a simulation on-the-fly of what’s around the robot and trying out many possible action plans. Vision models help us create a very realistic digital world that enables the robot to reason about feasible actions for each step of a long-horizon plan.”

The team’s work was presented this past month in a paper shown at the Conference on Robot Learning (CoRL) in Munich, Germany.

Play video

Teaching a robot its limits for open-ended chores MIT CSAIL

The researchers’ method uses an LLM pre-trained on text from across the internet. Before asking PRoC3S to do a task, the team provided their language model with a sample task (like drawing a square) that’s related to the target one (drawing a star). The sample task includes a description of the activity, a long-horizon plan, and relevant details about the robot’s environment.

But how did these plans fare in practice? In simulations, PRoC3S successfully drew stars and letters eight out of 10 times each. It also could stack digital blocks in pyramids and lines, and place items with accuracy, like fruits on a plate. Across each of these digital demos, the CSAIL method completed the requested task more consistently than comparable approaches like “LLM3” and “Code as Policies”.

The CSAIL engineers next brought their approach to the real world. Their method developed and executed plans on a robotic arm, teaching it to put blocks in straight lines. PRoC3S also enabled the machine to place blue and red blocks into matching bowls and move all objects near the center of a table.

Kumar and co-lead author Aidan Curtis SM ’23, who’s also a PhD student working in CSAIL, say these findings indicate how an LLM can develop safer plans that humans can trust to work in practice. The researchers envision a home robot that can be given a more general request (like “bring me some chips”) and reliably figure out the specific steps needed to execute it. PRoC3S could help a robot test out plans in an identical digital environment to find a working course of action — and more importantly, bring you a tasty snack.

For future work, the researchers aim to improve results using a more advanced physics simulator and to expand to more elaborate longer-horizon tasks via more scalable data-search techniques. Moreover, they plan to apply PRoC3S to mobile robots such as a quadruped for tasks that include walking and scanning surroundings.

“Using foundation models like ChatGPT to control robot actions can lead to unsafe or incorrect behaviors due to hallucinations,” says The AI Institute researcher Eric Rosen, who isn’t involved in the research. “PRoC3S tackles this issue by leveraging foundation models for high-level task guidance, while employing AI techniques that explicitly reason about the world to ensure verifiably safe and correct actions. This combination of planning-based and data-driven approaches may be key to developing robots capable of understanding and reliably performing a broader range of tasks than currently possible.”

Kumar and Curtis’ co-authors are also CSAIL affiliates: MIT undergraduate researcher Jing Cao and MIT Department of Electrical Engineering and Computer Science professors Leslie Pack Kaelbling and Tomás Lozano-Pérez. Their work was supported, in part, by the National Science Foundation, the Air Force Office of Scientific Research, the Office of Naval Research, the Army Research Office, MIT Quest for Intelligence, and The AI Institute.

[sleepisfortherich]

Felony or No Felony, WE DONT DO STASH SPOTS!!!! It’s better to Bond Out on a Possession Charge than Have Your people Missing You... #CourtRoomsOverFuneralHomes #RatherBeCaughtWithItThanWithOutIt #GetOnCode #LLM3 #SleepisForTheRichGunClub (at Shreveport, Louisiana) https://www.instagram.com/p/CPBE_dzBiQR/?utm_medium=tumblr

[meltedhorror]

“So.... You ignore everyone you like?”

Sixten rested his arms on the table in front of him, leaning forwards and smirking at the magenta ghost sitting in front of him. Micke looked up with a puzzled expression, responding with a silent “Uh.... No?..”

“Hm, alright...” The radio ghost nodded, faking a serious expression “So only those you love then?”

Sixten couldn��t help but grin at the others sudden change on posture, as Micke turned from his normal pale shade to a deep purple, the blue flames around him intensifying for a few seconds. He had tensed up and looked as if he just wanted to disappear beneath the table, and he probably would’ve if given the chance.

“These things better be turned off!” The animator sputtered out as he somewhat aggressively pointed to the microphone closest to him, gaining a VERY amused smile from Sixten as he just calmly shook his head. “A- And I’m not ignoring anyone for that matter!”

“You suuuure about that? Come on! I KNOW you got something going for that director!”

“N- no! Absolutely not!”

“Final offer?”

“There’s nothing going on alright-”

“So if I toooold hi-”

“Don’t you dare say anything!” Micke interrupted, shooting up from his seat and leaning across the table, looming over Sixten. But despite trying to come off as intimidating, the lime spectre couldn’t help but laugh at the pure flustered state that he was in.

“Final offer it was! You DO like him!”

At this point the animator looked like he was in a mixture of shock and anger - which he probably was, but most of all embarrassed. Well shoot, he officially outed himself now.

He sunk back into his seat, bending against the table and letting his elbows rest on it, then hiding his tired looking face in his hands. He exhaled shakily, letting his tail curl up beneath him.

“... Alright.... Alright, alright, maybe... B- but so what?”

“So what? You should tell him, that’s what!”

Now staring up at a very smug Sixten with a look of pure terror, he frantically shook his head. “No! No! Nu-uh! Never! Nope!”

At this point it was Sixtens turn to float up from his seat energetically, pointing to Micke and with a look of his own of pure chaotic energy.

“I’m a ghost Micke! But I’m not blind OR deaf for that matter! I’ve SEEN how you swoon over the man! And not only once or twice, no no! YOU my friend are in LOVE! And you have been for a long time! Correct?”

“W- Well....” His face somehow darkened even more than before. He couldn’t get a single word out at this point, so he just shyly nodded in response.

“See? I knew I was right!”

“Wait, you only GUESSED?”

“Not entirely, but sure! I guessed~” Sixten grinned widely, resting his hands on his hips “But let me tell ya, you’re as easy to read as a book! Just as, I say! I just wanted you to say it outright!”

“Alright alright, that’s it...” Micke got up and floated back a bit, obviously having had enough of this conversation and looking even more of a mess than usual in this wildly flustered state, but before disappearing into the floor he got out a sharp “If any of this is recorded I’ll make sure you’ll have a not so fun time!”

“Whatever you say Micke, whatever you say~”

[mindofkennedy]

k!llm3

[broomclosetbrew]

Trying out the Fiddler Unison Bourbon from ASW Distillery in Atlanta. This bourbon is a marriage of a high wheat and a high malt bourbon, one sourced and one distilled at ASW. This is an interesting concept but I don't know that it was successful just yet. Light on flavor, sweet, there are flavors of light caramel, grass, and wood. It is hot for 90 proof giving away its youth. This has potential and I'd be interested in trying their bourbon unmarried or this after it sat in a barrel a little longer. Paired with an Asylum Straight Jacket cigar which is bringing more of the wood notes out in the bourbon. .... "Age is an issue of mind over matter. If you don't mind, it doesn't matter." ~ Mark Twain .... #cigar #CigarLover #CigarAfficionado #cigarphoto #cigarsnob #cigarlife #cigarsmoke #asylumcigars #sailorandsticks #cigarsandwhiskey #TapThatAsh #whiskey #bourbon #aswdistillery #whiskeylife #WhiskeyLover #whiskeyphoto #WhiskeyNeat #WhiskeyAfficionado #whiskeygram #quote #quotestagram (at Coastal San Pedro) https://www.instagram.com/p/Cc6z_V-LLm3/?igshid=NGJjMDIxMWI=

[neritosam]

Uma mulher demais Ela é cheia de vida Uma pessoa sensível e dedicada. Busca, do seu jeito, fazer a diferença. Sempre honrando seus ancestrais. Sempre seguindo seus ideais. Sava, você é demais! ❤️ https://www.instagram.com/p/CORYNN-llM3/?igshid=jcr6ugarwomk

[angelievable-blog1]

at CvSU Main Campus, Indang https://www.instagram.com/p/B2Tm9N-llM3/?igshid=154l3jw1wqhsv

[onemagicaljourney-blog]

Dinner on boat in Bangkok. (på/i Bangkok, Thailand) https://www.instagram.com/p/Bnimbw-lLm3/?utm_source=ig_tumblr_share&igshid=17komk74q38pc