Organizing projects for optimal resource utilization and delivery time

There are several factors that contribute to performance and delivery time when it comes to rendering a project. The first one is, obviously, the hardware used for rendering. This is what we do: our system does its magic, and we make sure that the entire infrastructure always works at peak performance. And there are other factors that are related to the project that's being rendered - how well it's optimized and how it's structured.

This article talks about the last part - the project structure and more specifically, to the way the projects that are uploaded to our farm are structured. As projects grow and their disk space footprint is measured in GBs, hundreds of GBs or TBs, the project size starts to make a difference in storage space, upload and download time and even in asset transfer time from our storage system to the render nodes. Here are some recommendations that will help in the general project economy:

1. Organizing your RenderStreet FTP / uploads folder

We offer several ways to transfer the files to our system: you can upload a blend file or an archive on HTTPS directly to our site, or you can use FTP to upload an entire project, or you can transfer them directly from your cloud account , or you can use our plugin . Regardless of the upload method, all the files get to the same place in your account and you can see them in the FTP window from the 'Add render job' page. 

Keeping your uploads folder 'in shape' is important both in terms of total storage space used and in terms of rendering performance - especially for the RenderStreet One monthly plan. As this plan is best-effort, keeping unnecessary assets in the project folder can impact the total delivery time in a negative way. The examples below will give you a bit more insight into what our system does in the background.

a. The 'dump-everything-in-a-folder' structure

As far as optimization and performance go, this is the least efficient way of organizing the projects and we recommend avoiding it. 

In the example above, the 'ALL-PROJECTS' folder contains everything the user has uploaded. As our system always transfers the top-level folder to the render nodes, this means that regardless of the project the user is currently rendering - PROJECT-A, or PROJECT-B, or just the packed 2.blend file, or the ModoPed project, the  *entire* ALL-PROJECTS folder (colored in blue) will be transferred each time. So, even if the '2.blend' file, that is selected for rendering now, has only 683 KB in size, the entire folder totaling 6 GB (in this example) will be transferred. 

This makes for a very high overhead and * should be avoided at all times* . Each individual project should have its own folder in the FTP root directory, so that only the assets for that particular folder are transferred. This also makes managing projects easier for you, as you can track the projects folder easier

b. The multiple scene/shots project structure

In this scenario, a project has several shots, each shot with its own blend file(s). Al the shots share a common asset repository (textures, libraries, etc).

As you can see, the project is correctly placed in its own folder at the top of the FTP root directory. In this case, to keep the size down, we recommend rendering one shot at a time and keeping in storage only the files needed for the respective shot. So, in the example above, shot 1 is now being rendered. So the files in the folder should be the 'SHOT-1' folder and the 'COMMON-ASSETS' folder. The 'SHOT-2' folder is not needed now, so it wouldn't be uploaded as long as shot 1 is rendering.

When shot 1 has been completed, the 'SHOT-1' folder can be deleted and the 'SHOT-2' folder can be uploaded. Then the process repeats for each shot and/or scene in the project.

c. The project-in-a-single-file structure (packed/bundled file)

In this case, the project consists of only one main file, which can optionally have an additional assets folder. 

The recommended structure is, again, placing the file (and the assets folder, if applicable) in its own top-level folder. Alternatively, if the project only contains a single file, without any dependencies, it can be placed directly in the FTP root directory.

d. General considerations and RenderStreet One project size warning

As a project goes through several revisions, the number of files in the project folder might grow. In addition to the folder structure recommendations described above, we recommend these supplementary checks to keep the project trimmed:

- if you have set up your environment to create backup files automatically (e.g. 'blend1' files for Blender), set the backup folder somewhere else. If that's not possible, remember to not upload the backup files to our storage - they play no role in the rendering process

- if your folder contains bake caches that are not necessary to render the project, don't upload them to our storage.

- same observation for sound files, they are not needed when we render the project, so don't upload them

- if you have several revisions of the file(s) in the folder - we all know about the 'project-final', 'project-final2', 'project-final-approved', 'project-final-approved2', and so on - please upload only the version that you will use for rendering

- as the project size matters more for the RenderStreet One plan, we put a warning in place to alert you when the size of the selected folder is over a certain threshold. The warning will appear in the second step of launching a job. If you see it, please try reducing your folder size using the methods described above. If that's not possible, please let us know if we can help.

2. Render job size (frame count)  

We generally recommend keeping the number of frames in a new job under 2,500, as a rule of thumb. So, for instance, if you have 5,000 frames to render, you can split the render in two jobs of 2,500 frames each. Doing this will  *not* impact the delivery time, and has a couple of advantages:

- you'll be able to download the resulting renders easier (smaller download size per each job)

- for the monthly plan (RenderStreet One) users, you'll see a longer preview of the movie 

- for the monthly plan users, if you queue the jobs, you'll be able to download and process the first part of the project while the second / subsequent one is rendering. So you'll save time at the bottom line this way

Important note: if you know that your jobs will generate very large outputs (over 500GB), please give us a heads-up so we can optimize the process

3. Output format

The selected output format has a significant influence over the resulted render size. As a rule of thumb, we recommend choosing PNG over TIFF in all cases, unless there is a very specific reason that makes TIFF mandatory. This is valid both for animations and for still images. A particular note for renders launched with the split option: in this case, choosing TIFF will generate a very large file and, because of the format limitations, the image re-composition may fail.

For easier download, remember that we have the possibility to push renders to your cloud account. Or you can set up automatic downloads on FTP.

4. Launching the renders

If your project folder is large (over 10 GB) and you need to launch several jobs from it, please proceed as following: Launch the first job and wait until the first frame starts processing (shows progress). After that, launch the rest of the jobs from the same folder. This will optimize the cache transfer and all the renders will start faster.

• Related Articles

• General tips for getting the best results with RenderStreet

  This article is a collection of tips and recommendations that will help you get the best results with RenderStreet - from the very first time! 1. For all new projects, do a test run first. The test run can be a couple of frames from the scenes you ...

• Rendering Blender projects that have high memory requirements

  In some cases, a project may require more memory than available on the machine it's rendering on. If you see a message indicating this for one of your RenderStreet jobs, there are a few solutions you can apply in order to complete the render: 1. ...

• Using RenderStreet One with projects that need baking

  The RenderStreet One plan doesn't support baking. However, projects that need to be baked can be rendered with RenderStreet One. The simulations need to be baked first, as described in this article. After that, there are two ways to upload the ...

• How frame grouping works

  When 'Frame grouping' is selected in the interface, the system allocates a sequence of frames to a render node, instead of a single frame at a time. Activating this feature can significantly decrease the render time (and costs) for projects under ...

• How to Render a Blender file on RenderStreet

  On RenderStreet, a render is called a 'job'. The name applies to both still images and animations, and this article will show you how you can create jobs on our farm. In order to launch a job, you will need a positive credits balance (for on-demand ...