Procedural generation of library components

I have a set of python scripts that I wrote a while ago to parse Xilinx pinout files and generate library components. I would like to investigate porting these over to Horizon EDA.

Previously I attempted to make them work with LibrePCB, however the LibrePCB on-disk library format ended up being a complete nightmare due to the use of UUIDs for everything, including file names. The only option that the devs recommended was to cache all generated UUIDs so the scripts could be re-run without changing all of the UUIDs, however, this results in files containing many thousands of UUIDs which IMHO is a waste of space and inappropriate to commit to source control.

Horizon EDA seems to have a much more reasonable library format that doesn’t use UUIDs for file names, but does use UUIDs extensively within the libraries. Has much thought been put in to procedural generation that doesn’t require caching UUIDs or parsing the entire library?

One option that might be worth considering is hashing. Instead of randomly generated UUIDs, generating UUIDs based on hashing symbol and pin names results in deterministic UUIDs that don’t need to be cached. It seems that there is a version 5 UUID that is designed for doing just that, and it’s designed to support creating a hierarchy of deterministic UUIDs. Creating a version 5 UUID requires a namespace UUID and an identifying string. Perhaps a hierarchy of version 5 UUIDs could be used here, starting with root = uuid5(nil, “horizon-eda”), then uuid5(root, “part”), etc. It might also make sense to replace all pin UUIDs with deterministic UUIDs derived from the pin number so that scripts don’t have to parse all target packages to figure out what the correct UUIDs are when generating parts.

Automated package creation is quite comfortable in horizon (in my opinion), make sure to have a look at the scripts in the horizon-pool.

You are on the right track with the idea (afaik). The idea is basically that you get deterministic UUIDs, how you do it is really up to you, as long as you won’t produce collisions afterwards. I used this for package generation:

package_uuid = str(uuid5(NAMESPACE_DNS, filename))
pkg["uuid"] = package_uuid

The filenames were quite long and windy, so I have reasonable confidence they won’t collide with anything later on.

pkg is a json template read into a dict, I set all the pkg fields and then write out new json files. Just be aware that dicts don’t preserve order, and you have to manually deepcopy them if you want to do multiple edits.

Edit:

If it helps here is the whole thing I used for the panasonic electrolytics, which was from before the time where scripts where included in the pool. I didn’t have a look into it for a while, but there might be interesting bits in there for you:

# -*- coding: utf-8 -*-
import json, sys, copy, re
from uuid import uuid5, NAMESPACE_DNS
from pprint import pprint
from solid import *
from solid.utils import *  # Not required, but the utils module is useful
import math, os
import tempfile
import subprocess
from radial_electrolytic_capacitor import construct_part, write_scadfile

class hashabledict(dict):
    # A hashable version of a dict
    def __hash__(self):
        return hash(tuple(sorted(self.items())))




if __name__ == "__main__":
    # Read the basic parameters of the parts to be generated from here
    with open('partdata.json') as infile:
        parts = json.load(infile)

    # Use this package as a template to be modified
    with open('5mm_diameter_11mm_height_2mm_lead_spacing/package.json') as infile:
        package = json.load(infile)

    # Use this file as a template for the base parts
    with open('base_part.json') as infile:
        base_part = json.load(infile)

    # Use this file as a template for the child parts
    with open('child_part.json') as infile:
        child_part = json.load(infile)

    # Each diameter value has a different lead space and lead diameter, use this
    # as a reference
    measurements_per_diameter = {
        5.0:  {"lead space": 2.0, "lead diameter": 0.5},
        6.0:  {"lead space": 5.0, "lead diameter": 0.5},
        6.3:  {"lead space": 2.5, "lead diameter": 0.5},
        8.0:  {"lead space": 3.5, "lead diameter": 0.6},
        10.0: {"lead space": 5.0, "lead diameter": 0.6},
        12.5: {"lead space": 5.0, "lead diameter": 0.8},
        16.0: {"lead space": 7.5, "lead diameter": 0.8},
        18.0: {"lead space": 7.5, "lead diameter": 0.8}
    }

    # Find the unique package used in the list of parts in the csv and put
    # together a dict describing the basic measurements
    unique_sizes = set()
    for part in parts["parts"]:
        size = {
            "diameter" : part["diameter"],
            "height" : part["height"],
            "lead space" : part["lead space"],
            "lead diameter" : measurements_per_diameter[float(part["diameter"])]["lead diameter"]
        }
        unique_sizes.add(hashabledict(size))

    print("Found "+str(len(unique_sizes))+" unique package sizes")
    print()

    # Construct the parts and the part names
    for isize, size in enumerate(unique_sizes):
        print("-"*120)
        diameter = float(size["diameter"])
        height = float(size["height"])
        lead_space = float(size["lead space"])
        lead_diameter = float(size["lead diameter"])

        # Generate the variables used in the file name, avoid trailing zeros
        d = str(str(diameter)+"mm").replace(".0mm", "mm")
        l = str(str(height)+"mm").replace(".0mm", "mm")
        p = str(str(lead_space)+"mm").replace(".0mm", "mm")
        ld = str(str(lead_diameter)+"mm").replace(".0mm", "mm")

        # Construct parts
        part = construct_part(diameter, height, lead_space, lead_diameter)
        scad = "$fn = 64;\n\n"+scad_render(part)

        # Generate the filename and the path
        filename = "Polarized_Capacitor_"+d+"_diameter_"+l+"_height_"+p+"_"+"lead_spacing"
        path = os.path.join(".", filename+".scad")
        path = os.path.join("target", path)

        # Write the file to the path
        write_scadfile(path, scad)
        print("Wrote file "+path)


        # ====================== PACKAGE CREATION ==============================

        # Create a package
        pkg = copy.deepcopy(package)

        # For legacy reasons use this as the bassi for the hash:
        old_filename = filename.replace("height", "length")

        # Set deterministic UUID
        package_uuid = str(uuid5(NAMESPACE_DNS, old_filename))
        pkg["uuid"] = package_uuid

        pkg["tags"].append(p)

        # Set 3D model path with own deterministic UUID and delete the old one
        model = "3d_models/passive/capacitor/th/polarized/"+filename+".step"
        model_uuid = str(uuid5(NAMESPACE_DNS, model))
        pkg["models"][model_uuid] = package["models"]["844356f8-cb4f-4570-9ffc-cbc1629b33f8"]
        pkg["models"][model_uuid]["filename"] = model
        del pkg["models"]["844356f8-cb4f-4570-9ffc-cbc1629b33f8"]

        pkg["default_model"] = model_uuid
        
        # Set name for package.
        name = "Polarized capacitor, ⌀ {d}, height {l}, lead spacing {p}".format(d=d.replace("mm", " mm"), l=l.replace("mm", " mm"), p=p.replace("mm", " mm"))
        pkg["name"] = name

        # Clone and shift the pad 1
        n_id = str(uuid5(NAMESPACE_DNS, old_filename+"pad1"))
        pkg["pads"][n_id] = package["pads"]["3fb4c2a1-0bc7-4593-a31c-c19f2bdaa458"]
        pkg["pads"][n_id]["name"] = "P"
        pkg["pads"][n_id]["parameter_set"]["hole_diameter"] = int((lead_diameter+0.25)*1000000)
        pkg["pads"][n_id]["parameter_set"]["pad_diameter"] = int((lead_diameter+0.25+0.1+0.6)*1000000)
        pkg["pads"][n_id]["placement"]["shift"][0] = int(-1000000*(lead_space/2))
        del pkg["pads"]["3fb4c2a1-0bc7-4593-a31c-c19f2bdaa458"]

        # Clone and shift the pad 2 (same as pad 1, different UUID and different direction in the shift)
        n_id = str(uuid5(NAMESPACE_DNS, old_filename+"pad2"))
        pkg["pads"][n_id] = package["pads"]["8e435bbf-0e80-43fe-86cf-e4e4af12f4af"]
        pkg["pads"][n_id]["name"] = "N"
        pkg["pads"][n_id]["parameter_set"]["hole_diameter"] = int((lead_diameter+0.25)*1000000)
        pkg["pads"][n_id]["parameter_set"]["pad_diameter"] = int((lead_diameter+0.25+0.1+0.6)*1000000)
        pkg["pads"][n_id]["placement"]["shift"][0] = int(1000000*(lead_space/2))
        del pkg["pads"]["8e435bbf-0e80-43fe-86cf-e4e4af12f4af"]

        # Silkscreen arcs
        val = int(1000000*(diameter/2+0.2))
        pkg["junctions"]["d6b9aa8d-569b-4e31-af53-f41232ecc066"]["position"][0] = val
        pkg["junctions"]["42495c48-e83b-4378-b6c5-149bd0ef5657"]["position"][0] = -val

        # Silkscreen + symbol, set the position by using the percentual offset
        x = int((-(diameter/2*1000000)-250000)*0.8)
        y = int(((diameter/2*1000000)-250000)*0.8)+25000
        pkg["texts"]["c285ef7a-6dd9-4dbd-84c3-40df694c7bea"]["placement"]["shift"][0] = x
        pkg["texts"]["c285ef7a-6dd9-4dbd-84c3-40df694c7bea"]["placement"]["shift"][1] = y

        # Silkscreen $RD text, set the position by using the percentual offset
        x = -1376501
        y = int(-(diameter/2*1000000)-1200000)
        pkg["texts"]["05d9e4d9-32ae-4929-b74e-6edaf4a12214"]["placement"]["shift"][0] = x
        pkg["texts"]["05d9e4d9-32ae-4929-b74e-6edaf4a12214"]["placement"]["shift"][1] = y

        # Assembly $RD text, set the position by using the percentual offset
        x = int(-(diameter/2*1000000))
        pkg["texts"]["b6df677f-d236-471a-8b48-f63ddf8d6c9f"]["placement"]["shift"][0] = x


        # Offset the package outlines
        p_ids = [
            "645f1067-0745-47fc-a55d-2db940293bbf",
            "98613feb-3b6c-4dd4-9a87-ed0785fda52f",
            "c3f68824-13f4-4e3d-8948-af59fdf28cba",
        ]

        for p_id in p_ids:
            for i, v in enumerate(pkg["polygons"][p_id]["vertices"]):
                if v["position"][0] > 0:
                    delta = v["position"][0] - 2500000
                    new_x = int(1000000*(diameter/2))+delta
                    pkg["polygons"][p_id]["vertices"][i]["position"][0] = new_x
                else:
                    delta = v["position"][0] + 2500000
                    new_x = int(1000000*(diameter/2))+delta
                    pkg["polygons"][p_id]["vertices"][i]["position"][0] = new_x

            if p_id == "98613feb-3b6c-4dd4-9a87-ed0785fda52f":
                for i, v in enumerate(pkg["polygons"][p_id]["vertices"]):
                    if v["position"][0] > 0:
                        delta = v["position"][1] - 2500000
                        new_y = int(1000000*(diameter/2))+delta
                        pkg["polygons"][p_id]["vertices"][i]["position"][1] = new_y
                    else:
                        delta = v["position"][1] + 2500000
                        new_y = int(1000000*(diameter/2))+delta
                        pkg["polygons"][p_id]["vertices"][i]["position"][1] = new_y


        # Courtyard
        pkg["parameter_program"] = package["parameter_program"].replace("5.300", str(diameter))
        
        directory = os.path.join("packages", filename)
        outpath = os.path.join("packages", os.path.join(filename, "package.json"))
        padstacks = os.path.join("packages", os.path.join(filename, "padstacks"))

        if not os.path.isdir(directory):
            os.mkdir(directory)

        if not os.path.isdir(padstacks):
            os.mkdir(padstacks)

        with open(outpath, "w") as f:
            json.dump(pkg, f, sort_keys=True, indent=4)
            print("Wrote package for "+filename+" to ./"+outpath)


        # ======================= CREATE BASE PARTS ===========================
        
        bp = copy.deepcopy(base_part)

        base_name = "FC-A (Base part, ⌀ "+d+", height "+l+", lead spacing "+p+")"
        filename = "FC-A_series_diameter_"+d+"_"+l+"_height_"+p+"_lead_spacing_base_part.json"

        old_base_name = "FC-A (⌀ "+d+", Length "+l+", "+p+" Lead spacing, Base part)"

        # Set deterministic UUID
        bp_uuid = str(uuid5(NAMESPACE_DNS, old_base_name))
        bp["uuid"] = bp_uuid

        bp["tags"].append(p)

        bp["models"] = model_uuid
        bp["package"] = package_uuid

        bp["MPN"][1] = base_name

        outpath = os.path.join("FC-A_series", filename)

        with open(outpath, "w") as f:
            json.dump(bp, f, sort_keys=True, indent=4)
            print("Wrote base part for "+filename+" to ./"+outpath)


        # ======================= CREATE CHILD PARTS ===========================

        # Find all parts that use that package/base part

        ps = [part for part in parts["parts"] if (
            part["diameter"]==diameter and 
            part["height"]==height and 
            part["lead space"]==lead_space)]

        print("\n    Found "+str(len(ps))+" parts fitting for "+base_name+"\n")

        temp_mpns = []

        for i, part in enumerate(ps):
            if not part["mpn"] in temp_mpns and not part["mpn"].endswith(("B", "H", "E")):
                cp = copy.deepcopy(child_part)

                value = str(part["value"]).replace(".0", "")
                voltage = str(part["voltage"]).replace(".0", "")
                tolerance = str(part["tolerance"]).replace(".0", "")
                endurance = str(part["endurance"])

                child_name = part["mpn"]+"("+value+"μF,"+voltage+"V, "+endurance+"h)"
                child_mpn = part["mpn"]
                child_filename = part["mpn"]+".json"

                cp["MPN"][1] = child_mpn
                temp_mpns.append(child_mpn)
                cp["uuid"] = part["uuid"]
                cp["value"][1] = value+" μF ("+voltage+"V)"

                # Add Tags
                cp["tags"].append("passive")
                cp["tags"].append("th")
                cp["tags"].append("capacitor")
                cp["tags"].append("electrolytic")
                cp["tags"].append("polarized")
                cp["tags"].append(p)
                cp["tags"].append(voltage+"V")

                # Add parametric data
                cp["parametric"] = {
                    "table": "polarizedcapacitors",
                    "type": "Aluminium",
                    "tolerance": str(tolerance),
                    "mountingtype": str(part["mounting type"]),
                    "diameter": str(part["diameter"]),
                    "height": str(part["height"]),
                    "esr": str(part["esr"]),
                    "ripplecurrent": str(part["ripple current"]),
                    "tandelta": str(part["tan delta"]),
                    "mintemperature": str(int(part["min temperature"])),
                    "maxtemperature": str(int(part["max temperature"])),
                    "endurance": str(int(part["endurance"])),
                    "value": "{value}e-06".format(value=str(float(value))),
                    "wvdc": "{voltage}".format(voltage=voltage)
                }

                cp["model"] = model_uuid

                cp["description"][1] = "{value} μF, {voltage}V, {tolerance}%, {endurance}h Aluminium Electrolytic Capacitor".format(value=value, voltage=voltage, tolerance=tolerance, endurance=endurance)

                outpath = os.path.join("FC-A_series", child_filename)

                with open(outpath, "w") as f:
                    json.dump(cp, f, sort_keys=True, indent=4)
                    print("    ["+str(i)+"] Wrote child part for "+child_filename+" ("+cp["MPN"][1]+", "+value+"uF, "+voltage+"V, "+endurance+"h) to "+outpath+", tolerance: "+cp["parametric"]["tolerance"])

And the 3D-models have been created with this:

# -*- coding: utf-8 -*-
from solid import *
from solid.utils import *


def create_pins(spacing=5.0, diameter=0.5, height=6.0, cutdiameter=0.0):
    """
    Create the pins (spacing along the height side)
     ____________
    |            |
    |_          _|
    )___      ___(
    |            |
    |            |
    |____________|
      ||      ||
      ||      || ← Pins
    """

    pin1 = translate([(spacing/-2.0)-(diameter/2.0), diameter/-2.0, height*-1.0])(cube([diameter, diameter, height+cutdiameter]))
    pin2 = translate([(spacing/2.0)-(diameter/2.0), diameter/-2.0, height*-1.0])(cube([diameter, diameter, height+cutdiameter]))
    pins = pin1 + pin2
    pins = color([0.85,0.85,0.85,1.0])( pins )
    return pins


def create_body(diameter, height):
    """
    Create the body of a electrolytic capacitor (without legs)
     ____________   ← rounding
    |            |
    |_          _|  ← main_cylinder
    )___      ___(  ← donut_cutout
    |            |  ← (inside) inner_cylinder
    |            |
    |____________|  ← lower_rounding
      ||      ||
      ||      ||
    """

    # Variables
    segments = 64
    rounding = 0.5
    lower_rounding=0.5
    stripe_width = diameter*0.5

    # Upper donut
    extrude_circle = translate([diameter/2-rounding, 0, 0])(circle(r = rounding))
    donut = rotate_extrude(segments=segments)(extrude_circle)
    donut = translate([0,0,height-rounding])(donut)
    
    # Lower Donut
    extrude_circle = translate([diameter/2-lower_rounding, 0, 0])(circle(r = lower_rounding))
    lower_donut = rotate_extrude(segments=segments)(extrude_circle)
    lower_donut = translate([0,0,lower_rounding])(lower_donut)
    
    # Donut Cutout
    extrude_circle = translate([diameter/2+(rounding/10*18), 0, 0])(circle(r = rounding*2))
    donut_cutout = rotate_extrude(segments=segments)(extrude_circle)
    donut_cutout = translate([0,0,height/3*2])(donut_cutout)
    
    # Create the outer body
    main_cylinder = cylinder(r=diameter/2, h=height-rounding-lower_rounding, segments=segments)
    main_cylinder = translate([0,0,lower_rounding])(main_cylinder)
    
    body = donut+main_cylinder+lower_donut-donut_cutout
    
    # Create the inner body
    inner_cylinder = cylinder(r=diameter/2-rounding, h=height-lower_rounding, segments=segments)
    inner_cylinder = translate([0,0,(lower_rounding+rounding)/4])(inner_cylinder)
    inner_cylinder = color([0.8,0.8,0.8,1.0])( inner_cylinder )
    
    # Take the intersection of the stripe and the body
    stripe_block = translate([0, -stripe_width/2, -1])(cube([50,stripe_width,height+1]))
    stripe = intersection()(stripe_block, body)-inner_cylinder
    stripe = color([1,1,1,1.0])( stripe )
    stripe = translate([-0.01, 0, 0])(stripe)
    body = body-stripe_block
    body = color([0.1, 0.1, 0.1, 1.0])(body)
    body = stripe+body+inner_cylinder
    return body


def construct_part(diameter, height, spacing=5, lead_diameter=0.5):
    """
    Create a combined part of Pins and Body
    """
    cutdiameter = 0.2
    body = create_body(diameter, height)
    pins = create_pins(spacing, lead_diameter, 2.0, cutdiameter)
    return body + pins


def write_scadfile(path, element):
    """
    Write a scadfile to a given path
    """
    with open(path, "w+") as f:
        f.write(element)


if __name__ == "__main__":
    import sys
    FREECADPATH = "C:\\Program Files (x86)\\FreeCAD 0.18\\bin" # path to your FreeCAD.so or FreeCAD.dll file
    sys.path.append(FREECADPATH)
    import FreeCAD

Makes sense; thanks for the scripts, I’ll take a good look at those.

I was thinking it might be a good idea to come up with a ‘standardized’ UUID hierarchy, at least de facto for use in generation scripts, if it doesn’t get baked into the GUI as well. Instead of using NAMESPACE_DNS, we can start with our own root like so:

root_ns = uuid5(UUID(int=0), "horizon-eda")

Then, you can use that to create namespaces for other things:

part_ns = uuid5(root_ns, "part")
package_ns = uuid5(root_ns, "package")
# etc.

For pins, there are a couple of options. Apparently pins only need to be unique within each package, so you could use a global pins namespace, and then all pin 1s would have the same UUID, regardless of anything else. Alternatively, you could use the package UUID as the pin namespace, and then you would have unique pin UUIDs per package, but you would be able to generate the pin UUIDs. The second option probably makes a good deal of sense for symbol pins, but symbol pins are likely to be pretty unique anyway.

1 Like

I think it makes sense to think about common problems and best practice’s together and maybe make this part of the Documentation.

The more we help here with a good, well thought out guide, the less work scripted parts will create in the long run.

Thinking about UUIDs should definitely be part of this as well IMO