Got the following email from Rithmic after two days of running Optimus Flow. I suspect that the application is downloading far more than I have viewable on my DOM, T&S and the tick chart that I’m running, or that it’s downloading full historicals every time I fire it up, and not caching. Has anyone run into this issue with Rithmic and Optimus Flow?
Blockquote
Rithmic’s history plant guardian has observed that your user id, *******@amp.com, has requested historical market data resulting in 20099911211 bytes transmitted to your computers since Rithmic’s systems most recent scheduled reboot. You are approaching your weekly allotment of historical market data
Once you exhaust your allotment of data for the week your access to historical market data will suspended for the remainder of the week.
Thanks for your question and welcome to the community forum.
We have ran into this automated email from Rithmic in the past with several other clients and it typically results from a few things. In my experience, this typically occurs when a user constantly leaves their platform open, when they are using and downloading large volumes of historical tick data, or any indicator/feature that runs off of tick data.
For example, VWAP and Power trades are two features that both use tick data to populate. If you have these features running constantly, it could lead to high levels of historical market data being downloaded.
I just spoke with our developers on this and they are compiling a list of indicators and features for us that strictly use tick data in high amounts so we can attempt to avoid this issue from occurring again.
If you don’t mind me asking, besides your DOM, T&S, and tick chart what other features do you have open/applied to your platform?
Thanks for your time and please let us know!
Jake
Optimus Futures Support
Hi, Jake! Thank you for looking into this. As a matter of fact, I started up Optimus Flow to answer your question and immediately got this email from Rithmic:
*****@amp.com has requested 40003313651 bytes of historical market data out of a limit of 40000000000.
At this point our system has automatically suspended all Historical Market Data Privileges for this user for the remainder of the week.
In addition to the DOM, I have a 900 tick chart covering 1 day, a 1HR chart covering 1 month, a 5 minute chart covering 1 month, and 3 time and sales windows. In the 1HR chart I have volume profiles for this week and last week.
Edit: I have been closing Optimus Flow after RTH end.
Hi, an update here – I just ran some very basic network analysis, and it appears that when I have my workspace set the way it is now, and no other applications doing anything, it’s downloading around 2.5MB/minute. I think that comes out to around 1.2GB per 8 hours.
This makes me think that 40GB that Rithmic says I have downloaded in about 3 days must be coming from a large download that happens under normal circumstances when I start the program, but I can’t test this theory because Rithmic currently has me blacklisted from historical data.
Is there caching of data that is supposed to be happening but isn’t due to the way their feed works?
Had the same issue pop up and this is most likely the culprit, because in my case it was having a daily 3-month chart open. In an introductory call, I bought it up with Jake and he mentioned something like all Rithmic data being tick-by-tick, so pulling one month worth of historical data by tick rather than OHLC every time you open the client (and even a new chart) is probably causing a massive chunk of data to be downloaded.
Totally wish as well that there was some sort of caching.
Yes, here’s a chart that kind of shows it from this morning. The steady state just of the program being open is ~250Kb/second download, and then when it’s closed and reopened (gap on the chart) it spikes to around 3MB/second for a while. It tails off only when I close everything except a single time & sales window. Before that I had a tick chart showing cluster profile, time and sales, and a 1 day 5 minute chart open. No VWAPs, no volume profiles.
It’s actually difficult to see exactly how much data is being downloaded live vs historical, because there are multiple connections to different IP addresses being opened. 377MB just today from a server owned by Hibernia, and 90MB from a server in Texas owned by Rithmic. If the former is live data and only the latter is counted against Rithmic’s historical data cap then this is fine, but it’s hard to tell since Rithmic doesn’t tell you how much you’ve used, just when you hit the cap.
We appreciate both of your excellent feedback and a deeper look into this.
Rithmic’s data cap has been a bit tricky for some users to navigate through. While other users never even touch this cap, some users hit it within almost one day’s worth of time.
We’ve had discussions recently with our developers about this issue and like I mentioned, we are working on compiling a list for our users of the features that are most “data-hungry” on Optimus Flow so our clients will have an easier time understanding why they might be triggering this automated data cap so quick.
From my previous discussions, we can determine that anything related to tick data and volume related indicators seem to be taking up the most resources. Tick charts loading high tick count intervals, VWAP, and Power Trades are a few to name currently.
Another thing to consider is that there are numerous other platforms that users have reported this same issue, Jigsaw and Bookmap are two that frequently have this issue as well. I don’t think it is a coincidence that the platforms catered around order flow, volume profile, and heat maps all have the same issue.
Logically thinking, this seems to be one of the major culprits here and I agree with you, having some sort of caching feature would be amazing to avoid this. I will try and approach this with our developers and see if it is possible to store historcal data locally on the user’s machine rather than redownload it every time the user opens the platform.
We have already made a few improvements revolving around tick data when it comes to download speed and effeciency, in turn this has helped with the data cap as well, but it’s apparent this is a consistent issue for some users despite the tick data improvements.
We’ll keep this thread updated with any improvements on this subject. We appreciate your patience and troubleshooting!
