HOUDINI - LOPs Submitter - Proper USD Authoring for Husk Rendering - GridMarkets Pipeline 2.0

HOUDINI - LOPs Submitter - Proper USD Authoring for Husk Rendering - GridMarkets Pipeline 2.0

Introduction

When the GridMarkets Submit > GridMarkets Controls > Project Management > Use Husk option is activated any jobs submitted will be run directly using Husk on the GridMarkets machines.  Here are some things to know when starting rendering:
  1. Houdini is never opened on our end when rendering with Husk.
  2. Testing rendering from Houdini is not a valid test as the Houdini UI does a LOT of prep work and fixing before starting rendering that you will need to do when submitting.
    1. Please see Additional Resources section of this document for a link to a proper testing procedure.
  3. Once the .usd file is written properly, the project becomes very portable and should be more reliable in creating expected outputs.  This can take some getting used to doing though.
There are some pros and cons to this process.

Pros

  1. Houdini is never opened
    1. Since Houdini is not opened, there is less overhead and more machine resources can be dedicated to the rendering process
  2. Short time to first pixel (fastest time to start rendering)
    1. If done correctly, all of the information for rendering is embedded in the .usd file which is submitted with the job from your machine, meaning nothing has to cook or prepare for rendering.
  3. Cheapest/Most Efficient way to render
    1. The above points mean that less machine time is dedicated to preparing for a render, so most of the time you pay for is spent specifically on calculating pixels.

Cons

  1. Houdini is never opened
    1. This is a double edged sword.  Houdini never opening means that your scene is never cooked.  If your .usd file is not written completely or correctly, then there will be missing things in your render.
  2. Steeper learning curve
    1. Hopefully this doc makes it a little shallower, but there is much more to consider and ensure is correct before submitting.
  3. Potentially long write times and additional local storage required
    1. When writing out the stage to disk for submission, some of the information will need to be rewritten and/or recached.  This will take time to accomplish that varies by scene complexity.
Below are specific nodes and settings to be aware of when setting up your scene.
Info
This article contains information discovered by us and users while submitting scenes.  The information is guaranteed to be incomplete and there are likely better ways to do it.  If you know of better or different ways to successfully build a USD scene in Solaris to submit to GridMarkets, please share with support@gridmarkets.com so we can add them to this document.

Nodes

USD ROP

The LOPs submitter's workflow was designed with users writing out their own USD files using a USD ROP LOP node rather than using the nested node.  The nested node is intended to be a quick patch job, but there are far too many parameters which may or may not be pertinent to users to promote.  With this in mind, you can write a scene out using a separate USD ROP node to take fine control over the settings and target the file using GridMarkets Submit > Husk Controls > Output File.  When using this workflow, ensure that Husk Controls > Force Write is turned off so you do not accidentally overwrite the targeted file.

It is also suggested to channel reference between USD ROP > Output File and GridMarkets Submit > Husk Controls > Output File so the linkage stays consistent while submitting.

SOP Import

To date, this is the most reliable way to bring information from SOPs into the Stage for submitting to GridMarkets.  That said, you do still need to do a few settings to ensure that all of the required data is written to disk properly before submission.

Layer Save Path

This parm MUST be activated and a path on disk MUST be set.  This can be as simple as $HIP/usd/file.usd.  So far, we have found that leaving the frame variable out leads to reliable results with animated caches as they will be guaranteed to be authored with time samples if you do not include the frame variable.  With it, we have not seen a guarantee that caches will progress frame to frame properly.

Time Dependency

We have found that ensuring that a SOP which is targeted is correctly marked as time dependent makes a HUGE difference in both write times and sizes for the .usd files written by the USD ROP node when using SOP Import.  If possible, move time dependencies into LOPs.  For instance, instead of animating an attribute to drive materials on incoming geometry, make a static group in SOPs then use that in a shader in LOPs to animate the material.  This way instead of having to store all of the time samples for the incoming geometry, just the shader is changed, which is much smaller and faster to process.

Scene Import

Scene Import nodes seem to work best for bringing in lights, cameras, or static geometry.  They seem pretty fickle when authoring time samples and should be avoided for those purposes.  The bonus is that you can import many Object level nodes at once.  Again use them for importing all of your static/non-time-dependent geometry and SOP Import for the time dependent stuff.

Sublayer

These work great for bringing in geometry which has been cached to disk using a USD Export SOP.  This can be another solid way to make sure caches are respected, though caching through the USD Export SOP may be less space efficient.  Mileage may vary depending on how skilled you are with creating the caches that way.

Reference

Similar to Sublayer.  It is less tested than the other nodes, but has been tested to work.

Settings

USD ROP

Flush Data After Each Frame

If you are writing a stage which uses large caches, turning on this option may be necessary.  If Houdini crashes while exporting, the likely culprit is that the caching process exceeded your available system memory.  This option on the USD ROP will help to alleviate that, but it might also cause file sizes to bloat to unexpected sizes since data will not be able to be shared across multiple frames.  

Extra Files > Flatten File Layers and Extra Files > Flatten SOP Layers

These may or may not be needed when submitting, but turning them on seems to provide an added layer of reliability to the renders.  If you have done all of the above and are still facing issues where things are not animating as you expect, turn them on.

Other Resources

Please see these other knowledge base articles for more information on debugging, extending, testing, or submitting from the new GridMarkets V2 pipeline:
  1. LOPs Submitter Technical Documentation - GridMarkets Pipeline 2.0
  2. HOUDINI - LOPs Submitter - Proper Local Testing Procedure for Husk Renders - GridMarkets Pipeline 2.0