LOPs Submitter Technical Documentation - GridMarkets Pipeline 2.0

LOPs Submitter Technical Documentation - GridMarkets Pipeline 2.0

Part of the design principles for the new submission pipeline is a focus on simplicity, but also the ability to integrate directly into a new or existing pipeline within an studio.  The code comes primed with the default values (if any) for nearly all husk arguments found in the official SideFX documentation.

General Process Overview

How submissions are parsed

The LOPs submitter parses information for a job's Husk arguments directly from three places in this specific order:
  1. The rendersettings prim defined within the USD Stage
  2. The Render Settings node which is being targeted by the Submission Tool
  3. The Submission Tool settings overrides
Argument values from subsequent parsing stages will overwrite any which are already set.  This behavior allows the Submission node's overloads to function properly.

The scene is parsed by searching for known static property names within the rendersettings prim.  These are pre-defined in a dictionary (which is editable) and correlate the husk argument with the property or parm name and the expected data type.  

Extending Functionality

The adding new functionality to the tool is accomplished by editing a the constants.py file found at $HOUDINI_USER_PREF_DIR/otls/gridmarkets_lib/py3/constants.py.  This file contains many functions that are used elsewhere and will be detailed later in this document.  For now, focus on the Husk object.
Editing functions within this script could adversely affect or break the new submission pipeline as a whole.  Unless instructed specifically by support, do not edit anything unless you are sure you know what you are doing.
If you are sure you know what you are doing, it is ill advised to edit the functions, but adding entries to the dictionaries and objects included in the script can extend the functionality of the pipeline immensely.  
Be aware that this file will be overwritten whenever the GridMarkets tools are updated, so if editing, be sure to keep a backup copy of both the original file and the edited file outside of the default directory.
If you extend the functionality in a way that would be generally useful for the community or feel your changes should be included in the default settings, we encourage you to reach out to Support to have the changes evaluated.

Adding new hydra render delegates

Currently, the only hydra delegate which is setup functionally is Karma, so we will use it as an example.  Redshift, Vray, and Arnold have had their dictionaries started, but we are still investigating the correct settings for each before we are confident saying that they are supported.

Below is the HUSK.karma class which defines what is being searched.
class HUSK:
    class karma:
        hydra = {
            "cpu"                   : "BRAY_HdKarma",
            "xpu"                   : "BRAY_HdKarmaXPU"
        }
        attributes = {
            'bucket-order'          :('karma:global:bucketorder', str),
            'bucket-size'           :('karma:global:bucketsize', int),
            'dicingcamera'          :('karma:global:dicingcamera', pathlib.Path),
            'convergence-mode'      :('karma:global:convergence_mode',str),
            'disable-lighting'      :('karma:global:disable_lighting',bool),
            'engine'                :('karma:global:engine',str),
            'image-mode'            :('karma:global:imagemode',str),
            'progressive-passes'    :('karma:global:progressivepasses',int),
            'lock-random'           :('karma:global:randomseed',int),
            'pixel-samples'         :('karma:global:samplesperpixel',int),
            'camera'                :('camera', pathlib.Path),
            'pixel-aspect'          :('pixelAspectRatio',float),
            'res'                   :('resolution',tuple)
        }
        parameters = {
            'output'                :('picture', 'unexpanded_path'),
            # 'camera'                :('camera', pathlib.Path),
            'renderSettingsPrim'    :('primpath', str),
           
        }

Render Delegates

The hydra delegates are defined using the dictionary set to the hydra variable.  The keys in the dictionary correspond to the lowercase value for whatever list is specifying which delegate to use.  In this case, the engine parameter of the Karma Rendersettings node defines the hydra delegate used.  When this is parsed, it will get the lowercase version of the hou.Parm.rawValue() function and use that as the key for extracting the correct hydra delegate name from the dictionary.  When adding a new renderer, be sure this parameter is correct as it will be used to call the renderer from Husk.
hydra = {
    "cpu" : "BRAY_HdKarma",
    "xpu" : "BRAY_HdKarmaXPU"
}

USD Properties

USD Properties are parsed with and correlated to husk arguments by the dictionary set to the attributes variable.  The key is the exact husk argument which will be set by the USD Property value, and the value of the key is a tuple containing two things.  The first entry is the exact USD Property name associated with the value, and the second is an expected data type which will be used to process the incoming data.  If a property is missing from the default list, just follow the structure described here and the new argument will be defined on submission.
Currently supported data types are:
  1. str
    1. String values are extracted using their lowercase values as no predefined argument values for husk have capital letters.
  2. int
  3. float
  4. pathlib.Path
    1. Used to capture a general path for any pathlike object.  This includes disk paths, scene paths, or USD stage paths.
  5. bool
  6. tuple
    1. Specific case used solely for the resolution property.
attributes = {
    'bucket-order': ('karma:global:bucketorder', str),
    'bucket-size': ('karma:global:bucketsize', int),
    'dicingcamera': ('karma:global:dicingcamera', pathlib.Path),
    'convergence-mode': ('karma:global:convergence_mode',str),
    'disable-lighting': ('karma:global:disable_lighting',bool),
    'engine': ('karma:global:engine',str),
    'image-mode': ('karma:global:imagemode',str),
    'progressive-passes': ('karma:global:progressivepasses',int),
    'lock-random': ('karma:global:randomseed',int),
    'pixel-samples': ('karma:global:samplesperpixel',int),
    'camera': ('camera', pathlib.Path),
    'pixel-aspect': ('pixelAspectRatio',float),
    'res': ('resolution',tuple)
}

Adding new LOP target node parameters

Karma does not need many parameters scanned, mostly just the output file and the specific prim being used for the render settings, but what is needed is captured in the parameters variable.  The structure is very similar to the properties.  Key is the husk argument, first value in the tuple is the parameter name on the target LOP node, second value is the expected type.
The values of the parm are pulled using a hou.Parm.rawValue(), then processed using types if needed.
The special type processing values are:
  1. pathlib.Path
    1. Expands the path and creates a pathlib.Path object which is used for further processing.  Use this if the path is a node reference, USD Stage path, or static disk file.
  2. 'unexpanded_path'
    1. Uses the expanded file path and filename retaining and $F or $F# variables in the string.
parameters = {
    'output'                :('picture', 'unexpanded_path'),
    # 'camera'                :('camera', pathlib.Path),
    'renderSettingsPrim'    :('primpath', str),
}

Adding new overrides

Overrides can be added to the GridMarkets Submit LOP by including them in the Husk Controls tab and adding the override parameters to the HUSK.submitter.parameters variable. Spare parameters MUST include a toggle corresponding to the value which is being overridden which is used to enable or disable the parameter via the "Disable When" feature in the parameters tab.  The naming scheme is parmname_toggle for the toggle and parmname for the parm.  If additional parms are needed for defining the values, they must all be included in the dictionary with their accompanying arguments and all be enabled/disabled by the same toggle.

The structure for this dictionary is that the key refers to the parameter name, while the first value of the tuple is the husk argument which is being overridden, and the second is the expected data type.  The expected data types supported are the same as the properties value above.  If there is a parameter which is unused, such as the complexity slider when the custom value is not being used, 

Adding entirely new renderers

Before adding support for a new renderer to the tool, please consult the GridMarkets supported versions page.  If the renderer is not on the list, please contact GridMarkets Support to request support on our end before expending the effort to add the new functionality.
When adding an entirely new renderer, the above steps all must be completed.  In addition to those, the tool needs to be informed of the new nodes which it must be looking for.  To do this, you must add them to the appropriate sections of the NODETYPES object. 
  1. Start by adding the the list of nodes which are added by the renderer to the LOPs section of the NODETYPES object. 
    1. Use the existing values as a guide in how to add them. 
    2. The variable values must either be a specific hou.NodeType or a list of hou.NodeType. 
  2. Next, add the value for the renderer to the type_dict variable in the general_type function. 
    1. Again, follow the examples for how to manage this. 
    2. Each value MUST be a flat list of hou.NodeType. 
    3. The key that is added to the type_dict MUST be the same as the object added to the HUSK object.  These two are correlated in the code. 
  3. Finally, add the parm name for the output file to the parm_dict in the output_parm() function. 
    1. The list must be a flat list, best derived from the value in NODETYPES.newrenderer. 
    2. This will allow the tool to know from what parm to derive the output file information.
Once the above is completed, as well as linking everything up with the HUSK arguments, everything will be able to be submitted to the GridMarkets cloud.