Offline LLM Robot “Socratic” Control: Multimodal UR5 Magic

Offline LLM Robot “Socratic” Control: Multimodal UR5 Magic

Introduction

Imagine telling a robot arm what to do—like “Pick the gray block and put it in the pink bowl”—and having it figure out the multi-step plan using its own local vision and language reasoning. That’s exactly the vibe behind new research on Multi-modal Interactive Control of Robotic Arm based on Offline Large Language Models, described in the paper available at arXiv:2608.08183.

The core idea is refreshingly practical: instead of relying on paid online assistants (think ChatGPT-style services), the system runs an offline open model—specifically ChatGLM2-6B—on local GPU hardware. Then it combines that with visual object detection and a robotic manipulation “executor” so the robot can convert your text instruction into working code actions. The research frames it as an approach to “embodied AI” that’s more stable, more flexible, and far cheaper to iterate on.

This work proposes the “Socratic Models-ChatGLM” pipeline. It’s inspired by a Socratic-style approach—ask, reason, then act—but engineered into a modular robotic system using PyBullet for simulation, ViLD for open-vocabulary object detection, ChatGLM2-6B for local language-driven code generation, and CLIPort for pick-and-place execution. The authors report strong performance on both single-instruction multi-step tasks and continuous long-horizon multi-task sequences on a simulated UR5 robotic arm.

Why This Matters

This research is significant right now because robotic control is still one of the biggest “integration tax” areas in AI. Even when you have a great language model, you still need: perception, planning, action code, safety checks, and the messy glue that turns “understanding” into “moving the gripper correctly.” A lot of current demos lean on online LLM APIs, which means your robotics lab becomes dependent on network reliability and ongoing costs.

Running an offline model changes the game in a concrete way: you can iterate rapidly without worrying about rate limits or billing surprises every time you regenerate a plan. That matters for anything you’d want to try today, like a small warehouse experiment where a robot sorts items by color/label, or a lab setup where a robot needs to repeatedly do “copy-edit” style tasks—move objects to specific zones, group by category, then stack or relocate—based on natural language commands from technicians.

And it’s not just “LLMs + robots” again. This paper builds on prior Socratic-style robotics approaches by swapping the online dependency for a locally deployed model and showing that the system remains competitive across longer sequences. It also leans into multimodality: the robot doesn’t just hallucinate; it uses vision detections to give the LLM grounding, then translates the LLM’s output into code policies that downstream vision-based manipulation can execute.

Offline LLM Deployment: How Local ChatGLM Gets “Robot-Ready”

The authors’ pipeline depends heavily on the ability to call a language model like an OpenAI-style API, but running entirely offline. They deploy ChatGLM2-6B locally using common tooling so the rest of the system can “talk” to it without being rewritten every time you swap models.

The practical deployment trick

ChatGLM2-6B is an open-source bilingual chat model (Chinese/English) with 6.2B parameters. The paper notes that with quantization (INT4), it can run on consumer GPUs with at least 6GB VRAM (at INT4).

Deployment steps (as described):
1. Download the ChatGLM2-6B model and demo code from GitHub/Hugging Face.
2. Run web_demo.py and cli_demo.py to validate that the model responds properly.
3. Deploy an OpenAI-compatible local API using FastChat and Autogen, so the robot stack can request text completion/code generation by calling a local server endpoint instead of a paid external service.

Why this matters: for robotics experiments, you want to generate plans many times while tweaking prompts, parameters (like max_tokens and temperature), and scenarios. Offline deployment removes the friction that slows down real engineering.

Token budget as a real constraint

The system caps LLM output using max_tokens = 50 in the examples described. That’s a key detail because robotic plans can get long—yet the paper shows the system can generate compact “code policies” for core actions and then allow downstream modules to finish the step cleanly.

Socratic Models-ChatGLM: The Multimodal “Ask → See → Write Code → Execute” Loop

The heart of the paper is how ChatGLM2-6B is woven into a multimodal control pipeline—rather than treated as a detached chatbot.

Think of it like this:
- Vision is the robot’s “eyes.”
- ChatGLM is the robot’s “planner that writes action code.”
- CLIPort is the robot’s “hands” (it learns where and how to pick/place based on visual context).
- The system stitches them together using text-conditioned code generation, inspired by Socratic-style reasoning.

Step 1: User instruction becomes a structured target

A user starts with a direct command like:

“Pick the gray block and place it on the pink bowl”

In the PyBullet scene, there are 4 blocks and 3 bowls with different colors. That means the robot has a limited vocabulary of objects—but the LLM doesn’t know them until vision tells it what’s present.

Step 2: ViLD creates an object list the LLM can trust

The paper uses ViLD (open-vocabulary detection) to find objects on the table and convert them into a list like:
objects = ["yellow block", "yellow bowl", "pink bowl", "pink block", "gray block", "brown bowl", "orange block"]

This is the grounding mechanism. Instead of making the LLM guess which objects exist, ViLD provides explicit candidate names so the LLM can write code referencing real entities in the scene.

Step 3: ChatGLM2-6B generates hierarchical code policies

Now the system calls the locally hosted ChatGLM2-6B API to produce an action plan in code form. The paper describes that the LLM generates steps as natural-language lines and/or pseudocode templates, eventually yielding something like:
- robot.pick_and_place("gray block", "pink bowl")

It also mentions that after generating the core execution line(s), the LLM can generate supplemental text (like continuing the objects list) depending on prompts and context. In practice, the important part is the core robotic policy instruction that downstream execution can run.

Step 4: CLIPort turns code policy into actual grasps

Finally, the code policy is passed into CLIPort, which uses pick-place heatmaps to choose grasp and placement points.

This is a crucial design choice: even if the LLM generates correct intent, the robot still needs a learned visual manipulation model to physically execute in a way that works with pose, clutter, and contact dynamics.

How this compares to earlier work

The paper positions itself relative to:
- Code as Policies (which uses LLM-driven code planning, but typically depends on online services).
- Socratic Models (the earlier Socratic robotics idea).
- Their own contribution: offline local ChatGLM2-6B + multimodal UR5 pipeline.

Here’s the high-level comparison in terms of the LLM deployment assumption:

(Details are drawn from the paper’s described method choices and motivation.)

What the Researchers Actually Measured: Task Scores, Completion Rates, and Execution Time

The experiments are set up using PyBullet simulations with varied block/bowl configurations (mostly color and arrangement). The goal is to see how well the whole pipeline works—especially whether the LLM-generated code policies are coherent enough for the robot executor to do the right thing.

Evaluation criteria used in the paper

The authors evaluate performance with a score based on:
- Accuracy of control code policies generated by ChatGLM2-6B
- Whole completion time and whether there are errors or redundant actions during execution
- Consistency between the final result and the user’s text demand

They also report qualitative results and task completion behavior, and they mention recording videos for continuous multi-task experiments.

Single-instruction multi-step tasks (3 tested scenarios)

They report three main single-instruction multi-step tasks:
1. Sort the blocks (“Sort the blocks in the way you think it best fits.”)
- 4 bowls + 4 blocks
- Completed with 100% task completion rate
- Reported time: 2 mins 42 s
- Performance score: 88%
- Note: the score is limited because ViLD makes some recognition errors for pink and yellow blocks.
2. Move blocks to the corners
- 4 different colors (blocks/bowls)
- Completed in 3 mins 16 s
- Task completion rate: 100%
- Performance score: 88%
- Note: less “flexible” placement—placing uniformly in the upper left rather than distributing across distinct corners.
3. Move blocks in the middle
- Sequentially place the purple block and yellow block into the middle area
- Completed in 3 mins 11 s
- Task completion rate: 100%
- Performance score: 85%
- Note: redundant action happened—after placing yellow in the middle, it got moved back to the left bowl before the final state.

A practical interpretation of these results

These numbers tell a specific story: the system is often good enough to achieve the right final arrangement (100% completion), but the journey matters too. Performance score drops when:
- perception (object detection) is slightly off,
- the LLM code plan is less sensitive to geometric nuance,
- or the pipeline includes redundant steps.

That’s honestly what you’d expect in early embodied systems. The interesting part is that offline ChatGLM2-6B still delivers high completion reliability while enabling extensive experimentation without online cost constraints.

Long-Horizon Continuous Tasks: When Planning Has to Stick Across Multiple Commands

The paper doesn’t stop at single-shot missions. It tests continuous multi-task sequences where a user gives multiple commands in a row, and the system must keep updating based on the evolving scene.

Why offline LLM helps here

A major point the authors make: because ChatGLM2-6B runs locally, it’s relatively cheap to call repeatedly. That matters more as tasks get longer, because you need more planning calls across steps.

Continuous task types demonstrated

They describe multiple continuous scenarios (labeled Task A, B, C, D) and report results at the level of completion and scores.

Continuous Task A: 3 consecutive missions

The sequence:
1. “Pick the gray block and place it on the pink bowl.”
2. “Move the yellow colored block to the bottom side.”
3. “Put all the blocks to the top right corner.”

Reported outcomes:
- Task A-2: completes in 1 min 23 s, performance score 82%
- Task A-3: 100% completion, performance score 86%

The pipeline adapts to the next instruction by again using ViLD and then regenerating code with the offline LLM.

Continuous Task B: color grouping and matching

They test two missions:
1. “Group the pink objects together.”
2. “Match the blocks with similar-colored bowls.”

Reported outcomes:
- Second mission: 100% completion and 92% performance score

Continuous Task C: move then re-sort

They describe:
1. “Move the blocks to the bottom side.”
2. “Sort all the blocks into their matching color bowls.”

Outcome highlights:
- The paper claims this illustrates flexibility—especially benefiting from ViLD detecting objects effectively after the scene changes.

Continuous Task D: four long-horizon steps including small displacements

Task D is the most “long-horizon” style described:
1. Move the gray block left about 3 cm
2. Place the yellow block bottom down about 4 cm
3. Stack blocks on the pink bowl “as possible”
4. Pick the cyan block to the upper right corner

The paper notes:
- Task D-1 and D-2 are designed to test controlled relative displacements and report performance comparable to an online reference reported in Code as Policies (they mention “equal excellent performance” in this context).
- The later stacking/placing steps work well enough to complete successfully, with the key advantage attributed to the offline model’s flexibility and reusability.

What this means for real robots

Long-horizon control is where systems often fail: small execution errors compound, perception drifts, and planning becomes inconsistent. This paper’s approach tries to reduce that by:
- grounding each step using updated ViLD object detection,
- regenerating code after each user instruction,
- and relying on CLIPort to handle the visual manipulation details.

Practical Implications: How You Could Use This Approach Today

If you’re building or prototyping embodied AI, there are a few practical lessons buried in the engineering choices.

1) Treat the LLM as a code-writing “planner,” not the robot itself

The system doesn’t ask the LLM to directly “control physics.” Instead, it uses the LLM to output a compact policy (e.g., robot.pick_and_place(...)). That’s a safer and more controllable division of labor.

2) Ground everything with a vision-generated object list

Using ViLD to generate an explicit objects = [...] list reduces the chance that the LLM refers to objects that don’t exist—or confuses categories. Even when ViLD errs (as seen in the 88% vs 100% scoring gap), it’s still the LLM reacting to the current state rather than blind guessing.

3) Offline deployment makes experiments scalable

Because there’s no need for ongoing paid API calls, you can afford many trials. The authors specifically highlight being able to repeatedly apply the model to run lots of experiments across task types. That’s a huge advantage during development, prompt tuning, and ablation-style testing.

4) Expect “completion” and “score” to diverge

The paper reports cases where completion is 100% but performance score drops due to redundant actions. In real deployments, you might care about:
- speed,
- minimal movements,
- energy consumption,
- and reducing wear on mechanisms.
So the work suggests the next step is improving plan efficiency and reducing unnecessary steps.

If you want the full project materials, the authors point to a GitHub repo: https://github.com/2000222/Socratic-Models-ChatGLM

Key Takeaways

• Offline local LLMs can work well for robotic control: using ChatGLM2-6B deployed locally (OpenAI-compatible API) avoids online cost and key dependency.
• The “Socratic Models-ChatGLM” pipeline is modular and multimodal: ViLD (vision grounding) → ChatGLM2-6B (code policy generation) → CLIPort (pick-and-place execution).
• Single-instruction multi-step tasks achieved strong results: reported task completion 100% with performance scores around 85%–88% and execution times from about 2:42 to 3:16.
• Long-horizon continuous tasks also work, including sequences up to 4 consecutive missions, demonstrating the system can keep adapting as the scene changes.
• Performance score penalizes inefficiency and perception errors: even with 100% completion, redundant actions and ViLD misrecognitions can reduce the score (e.g., 85% when redundant movement occurred).
• Next frontier is efficiency and robustness: completion is reliable, but reducing redundant steps and improving grounding accuracy (so fewer score losses happen) are natural next improvements.