It turns out that LLMs can make CAD models for simple 3D mechanical parts. And, I think they’ll be extremely good at it soon.
See discussion on HackerNews
GitHub Repo for Text-to-CAD Evaluation
Results dashboard
An AI Mechanical Engineer
Code generation is the first breakthrough application for LLMs. What would an AI agent look like for mechanical engineering? Material selection, design for manufacturing, computer-aided manufacturing (CAM), and off-the-shelf part comparison would all be important features of an AI mechanical engineer. Perhaps, most importantly, an AI mechanical engineer would design and improve CAD models. Mechanical engineers typically design CAD using point-and-click software (e.g. Fusion 360, Solidworks, and Onshape). How could AI generate these solid models instead?
Code generation meets CAD
One promising direction is training a generative model on millions of existing CAD files. This approach is being actively researched by multiple teams who are investigating both diffusion and transformer architectures. In particular, I like Autodesk Research’s approach to encode the parametric primitives (points, curves, shapes, extrusions, etc) into a transformer architecture. However, as far as I understand, the models in these projects cannot yet take an arbitrary input command and generate a desired shape.
Then a few weeks ago, I was inspired by the recent use of LLMs to drive Blender, the open source modeling tool widely used for animation. Given that LLMs are incredibly good at generating code, perhaps programmatic interfaces for CAD modeling could be used to generate solid models in a similar way. I immediately thought of OpenSCAD, an open-source programmatic CAD tool that’s been developed for more than 15 years. Instead of using point-and-click software to create a solid model, the user writes a software script, which is then rendered into the solid CAD model.
LLMs rock at writing OpenSCAD
To test it out, I created a simple project in Cursor, made a blank OpenSCAD script (Cursor.scad), and added some Cursor rules:
# Your rule content
– We’re creating files to model things in open scad.
– All the OpenScad files you create will be in Cursor.scad. I’ve set up this file such that if you edit it, it will automatically be read by OpenScad (it’s the open file in the program).
– If I want to save what you’ve done, I’ll tell you and you should create a new file and put it in the Saved folder.
– That’s it! Overtime, if needed, we could create documentation about how to use OpenScad.
– If I’m asking you to create a new design, you should delete the current contents of cursor.scad and add the new design into it.
– When I make requests you should always first develop a step by step plan. Then tell me the step by step plan. And then I’ll tell you to start modeling.
– When you’re going through the step by step plans, only execute one step at a time.
– When you’ve executed a step, ask the user if its right.
Then, I started using Cursor to create solid models.
Here’s an example: “Create an iPhone case”.
It didn’t nail it on the first try, but with a couple of iterations (including giving it screenshots) we created a basic case.
You can also leverage OpenSCAD libraries (there are many public ones). Here, I use a library to make a thread for a flange.
One thing that’s pretty neat is that the LLM can use its general knowledge of mechanical engineering. For example, above, Cursor created holes in the pipe for M6 bolts and it correctly made the diameter slightly bigger than 6 mm, so the bolts could pass through.
bolt_hole_d = 6.5; // Diameter for M6 bolts
Of course, one of the really nice things about this approach is that the files are editable and Cursor defaults to parameterizing all the key elements of the design. In the above example, I asked it to add holes for mounting bolts, which it did, and then I edited the number of holes manually to 3 from 4.
// Flange parameter
flange_OD = 50; // Outer diameter of the flange in mm
flange_thickness = 10; // Thickness of the flange in mm
pipe_size = 1/2; // NPT Pipe Size
// Bolt hole parameters
num_bolts = 3;
bolt_hold_d = 6.5; // Diameter for M6 bolts
bold_hole_circle = 35; // Diameter for the bolt circle
Building an eval for LLM -> OpenSCAD -> STL
I was impressed by these initial results but I wanted to learn more. For example, did the model’s reasoning ability help it think through the steps of creating a part? So, I decided to develop an evaluation to test the performance of various LLMs at generating solid models via OpenSCAD.
One of the challenges with creating an eval for CAD design is that most tasks have many correct answers. For example, a task such as “make a m3 screw that’s 10mm long” could have many correct answers because the length, diameter, and style of the head are not defined in the task. To account for this, I decided to write the tasks in my eval such that there was only a single, correct interpretation of the geometry.
For example, here is one of the tasks in the eval:
This is a 3mm thickness rectangular plate with two holes.
The plate is 18mm x 32mm in dimension.
When looking down at the plate, it has two holes that are drilled through it. In the bottom left of the plate, there’s a hole with a centerpoint that is 3mm from the short (18mm) side and 3 mm from the long (32mm) side. This hole has a diameter of 2mm.
In roughly the top left corner of plate, there’s a hole of diameter 3mm. Its center point is 8mm from the short side (18mm side) and 6mm away from the long (32mm) side.
The benefit of this approach is that we can score each task as a Pass or Fail and we can do this in an automated way. I wrote 25 total CAD tasks which ranged in difficulty from a single operation (“A 50mm long pipe that has a 10mm outer diameter and 2mm wall thickness”) to 5 sequential operations. For each task, I designed a reference CAD model using Autodesk Fusion 360 and then exported a STL mesh file.
Then, I set about programming the automated eval pipeline (of course, I didn’t actually write much code).
Here is how the eval pipeline works:
For each task and model, the eval sends the text prompt (along with a system prompt) to the LLM via API.
The LLM sends back the openSCAD code.
The openSCAD code is rendered into a STL
The generated STL is automatically checked against the reference STL
The task “passes” if it passes a number of geometric checks.
The results are then outputted in a dashboard.
graph LR
A[Start Eval For each Task & Model] –> B{Send System + Task Prompt to LLM};
B –> C[LLM Returns OpenSCAD];
C –> D{Render OpenSCAD to STL};
D –> E{Compare Generated STL to Reference STL};
E –> I[Output Eval Results to Dashboard];
[Note: The eval runs multiple replicates per task x model combo. And the eval is executed in parallel, because there can be 1000+ tasks when running the full evaluation.]
Here’s how the geometric check works:
The generated STL and reference STL are aligned using the iterative closest point (ICP) algorithm.
The aligned meshes are then compared by:
Their volumes (pass =
