Min 1 Minute interval between scans if data source
-
Hi there,
We’ve been testing Templater Bot. We like it so far but have found the 1 minute min interval for checking a local json file or URL data source to be quite limiting. We have been looking for ways around this as we would like it to be instant, or as instant as possible. We tried using a disable and re-enable of the bot to start a scan to solve this, but it seems it purposely waits 1 minute before doing the first scan.
What is the reason for this, and are there any solutions to it?
We greatly appreciate any help.
-
@Singit we’ve lowered the polling time to 30 seconds for the next release. Shorter polling than that can cause unexpected results or instability.
Also, if you architect your app to use the CLI instead of the Bot, you can “push” data into Templater, instead of “pulling” with the Bot. You could set up triggers with your app that launch Templater whenever data is added. Check out this thread for some basics.
-
Hi Jon,
Thanks for the quick response. That seems to be exactly what we are looking for. So if we run templater from the CLI and we use a JSON URL datasource, it will instantly poll that data source as soon as templater runs? Or would we need to use a local JSON datasource?
-
OK, we tested it and it seems it works both ways instantly, which is perfect. One last question, is it possible to run templater from the CLI without having to constantly open and close AE? I.e. Can we give templater render tasks from the CLI while keeping AE open?
-
@Singit Yes, as you discovered, it will immediately poll the data. I don’t believe that it’s possible to leave AE open without using the CLI to launch the Bot, which defeats the purpose of using the CLI for immediate polling of data. You will need to assess which is more important to you, and which saves you more time in your particular processing tasks.
-
Hi @Jon .
We’ve done a load of testing the last week. The CLI does actually work with AE open.
We have found an issue though, or at least what we consider an issue. After doing a ton of working, it seems that even on the CLI there is a large delay of about 20 seconds when getting JSON data from a URI. It seems the bot just hangs for no reason, since the data is there and it shouldn’t take longer than milliseconds to get it.
Can this be fixed?
-
@Jon After doing some further testing, it seems to be a bug, as this behaviour does not happen when you run the render with all the same settings (including the remote JSON source) from the GUI Templater. This only occurs when running Templater from the CLI.
-
@Jon After doing some further testing, it seems to be a bug, as this behaviour does not happen when you run the render with all the same settings (including the remote JSON source) from the GUI Templater. This only occurs when running Templater from the CLI.
EDIT: It definitely seems to be a bug. When replicating instead of rendering, pulling in remote JSON data takes less than a second. Only we rendering it takes up to 45 seconds to pull in the same data.
-
@Singit this is a known issue with the CLI and JSON polling intervals. We have addressed it, fixed it, and it is in the 2.9 release which we’ll announce tomorrow — Feb. 25, 2019. In that release, with the CLI, Bot’s polling rate can low as 0.5 minutes in the 2.9 release. Thank you for your patience as we make our product more robust.
-
@ariestav We actually aren’t using the ‘bot’ functionality. We are doing a simple ‘render’ through the CLI, so there should only be one instant poll. This works when you use the ‘replicate’ function, but not when you use the ‘render’ function.
-
@Singit Thank you for the clarification. Maybe AE is taking its time to prepare for actual rendering — loading output modules, render settings, etc.? Are you certain that the issue is related to data retrieval? We have noted the issue and will try to reproduce it.
I have seen cases where the CLI can take much longer to complete when rendering due to Templater making use of AE’s render engine. When you launch AE’s with its GUI normally, it’s likely that its rendering engine modules are loaded automatically, but via the cli it can feel like it’s the cause of delay. With replication, there is no use of AE’s rendering engine — the layers are invalidated with new data and then the project is copied to a new file, so it is speedier in execution.
What do the time stamps in
templater.logfile say? -
@ariestav I’ve been pretty careful to find where the bug lies. I’ve tested it out pretty much every which way possible from both the GUI and CLI. I’ve also checked the task manager to see the resource usage. It really just hangs doing nothing from 30-60 seconds. This is also shown in the log file. I just updated and it also still occurs in the updated version. We are running it with the GUI open so that it doesn’t need to load AE. But I did test without the GUI as well and it is the same.
Here is a tidied up and redacted log file…
2019-2-23 @ 15:26:44 - Session started 2019-2-23 @ 15:26:44 - Templater is searching for manufacturer's licensing framework. 2019-2-23 @ 15:26:44 - Initializing Templater CLI 2019-2-23 @ 15:26:44 - Starting Command Line Interface with options file. 2019-2-23 @ 15:26:44 - Main options log_location => C:\xxxxx data_source => https://xxxxx aep => C:\xxxxx source_footage => C:\xxxxx output_location => Y:\xxxxx render_settings => Yap output_module => Yap output_prefix => row_start => 1 row_end => 5 save_on_completion => false quit_on_completion => false quit_on_errors => tasks => [object Object] prefs => [object Object] 2019-2-23 @ 15:26:44 - Prefs options default_target => Icon_1 tsv_from_excel => false replica_comps_to_q => false replica_comps_to_a => false remove_replicas_am => false fix_layer_exp => false missing_footage => 3 placeholder_visual => C:\xxxxxx placeholder_audio => C:\xxxxxx unique_downloads => false suppress_ae_dialog => true silence_logging => false purge_caches => false remote_json => [object Object] bot => [object Object] 2019-2-23 @ 15:26:44 - Bot options name => kaori goog_freq => 5 json_freq => 1 max_queued => 5 action_render => true action_replicate => false pre_cmd_data => pre_jsx_data => pre_cmd_batch => pre_jsx_batch => pre_cmd_job => pre_jsx_job => pre_cmd_update => pre_jsx_update => pre_cmd_output => pre_jsx_output => post_cmd_data => post_jsx_data => post_cmd_batch => post_jsx_batch => post_cmd_job_async => true post_cmd_job => C:\xxxxxxx post_jsx_job => post_cmd_update => post_jsx_update => post_cmd_output => post_jsx_output => enable_cmd => enable_jsx => shutdown_cmd => shutdown_jsx => 2019-2-23 @ 15:26:44 - Remote JSON options username => password => headers => [object Object] 2019-2-23 @ 15:26:44 - Remote JSON headers 2019-2-23 @ 15:26:44 - Task options render => true replicate => false bot_enable => false 2019-2-23 @ 15:26:44 - WARNINGS regarding CLI options 2019-2-23 @ 15:26:44 - There is no specified output prefix. Consider adding an output prefix to group batch output together 2019-2-23 @ 15:26:44 - Examining URI https://xxxxxx 2019-2-23 @ 15:26:44 - Detected URL as a remote JSON resource https://xxxxx 2019-2-23 @ 15:26:46 - Configuring Templater's preferences to match values in templater-options.json file 2019-2-23 @ 15:26:46 - Setting Templater's preferences to match values in options file 2019-2-23 @ 15:26:46 - Executing tasks with the following settings 2019-2-23 @ 15:26:47 - Templater is executing requested tasks as specified in options 2019-2-23 @ 15:26:47 - Executing batch render of video output 2019-2-23 @ 15:26:47 - Data source is Remote JSON 2019-2-23 @ 15:26:47- Using start row 1 2019-2-23 @ 15:26:47 - Using end row 5 2019-2-23 @ 15:26:47 - Retrieving objects 1 through 5 from from remote JSON feed. Please wait... 2019-2-23 @ 15:27:26 - Now rendering target for rows 1 through 5 2019-2-23 @ 15:27:26 - Render job data 2019-2-23 @ 15:27:26 - Performing layout for 6 layers 2019-2-23 @ 15:27:26 - [ --__== DEPTH 0 ==__-- ] 2019-2-23 @ 15:27:29 - ---- RENDERING TARGET ---- 2019-2-23 @ 15:27:43 - ---- DONE RENDERING TARGET ---- 2019-2-23 @ 15:27:43 - Asynchronous command executing on [ bot_post_job ] event 2019-2-23 @ 15:27:43 - Completed rendering rows 1 through 5. -
Many thanks for these details. They will help us troubleshoot. It does seem strange that retrieval begins at
12:26:47, but then the next message that readsNow rendering target for rows 1 through 5is stamped 39 seconds after that @15:27:26. We will try to reproduce — are you using any kind of authentication scheme for the remote JSON feed?Thanks,
Arie -
@ariestav Yeh, it’s strange. The file being rendered is also very very simple. It normally takes about 1 second to render when done manually from the GUI.
We aren’t using any authentication on the remote JSON feed.
-
@ariestav Any update on the bug?
EDIT: It’s now becoming a bigger and bigger issue. It now hangs for about 2 minutes when pulling in data from the JSON source. It is getting to the point where we are looking at no longer using templater bot, as we haven’t had any indication this will be fixed.
