From: "Benson, Kevin" <k.benson@ucl.ac.uk>
Subject: Info on HKMonitor
Date: 25 August 2021 at 11:53:59 BST
To: "Whillock, Matthew" <m.whillock@ucl.ac.uk>

Hi Matt,
 
Below is some various information and suggestions plus where you might find where to look at in the code.  And placed a few of the main tasks below.
 
Tried to place everything I can think of down. The url links below should be without a password so you can easily download.  You can try with your desktop but if it gives you trouble we can use mssl1e with X2GO or VNC.  We can chat on Teams later if needed during this upcoming days.  I will be going on holiday for a week starting tomorrow but hope to check a few emails ever now and then, feel free to wait till I get back or have an attempt at a few things.  
 
**Interestingly it is not breaking like I recall from awhile back a least not with the CEA sample data.  Still a few issues to look at that I think you can solve faster than me. When it comes to GUI development, I do tend to struggle.   I will try this afternoon to recheck ART and EDDS data that I recall was giving me a headache.
 
Data:
I suggest we simply get things going with ‘CEA’ data that you did before and then we can move to the brand new EDDS data or possibly the ART Test data.  Simpler to do ‘CEA’ first since it was the original and really any issue fixed with it should just about fix other data.  I will send ART and most importantly EDDS data soon, the CEA data though you will find in the zip files.
 
HK Feed/Pipe:
We have two separate programs that will take in input data and write out a csv output file.
A.)       The one you or somebody originally created a long time ago to feed in one input file and writes it out streaming to an output file.  This is what you should use for the tests and is the simplest.
B.)       The other program is a java program that takes in many input files and tails/listens for any new data on those input files and merges them into one output file.  Ignore it though for these tasks.
For ‘A’ it can be ran  with this command.  (the pl file is the same one you had before just renamed). 
./HKFeedSingleFile.pl /home/iwsop/HKFeeder/CEA_Day3_3.25.csv /home/iwsop/HKFeeder/TestFeed2.csv
 
Or alternatively you will find in the HKMonitor  (2nd zip file link) directory various ‘feed’ shell/pl scripts that can be slightly edited, they are the same pl as the HKFeedSingleFile.
You can find the program here:
http://msslkz.mssl.ucl.ac.uk/usdownloads/euclid/HKFeeder.zip
 
HKMonitor.
Really has not changed much since you had it.  I tried to leave things as much as I could the same but I did add a few new features.
1.)       EchoReader.ini – Sort of important.  Has a ‘name’ of type of csv file to expect and the index columns where to find everything.  Hence will start the HKMonitor with ‘CEA_ORIG’.  I know another solution would be to try to make a converted csv with same column indexes but was worried that I would like to let HKMonitor quickly adapt to any new CSV file coming out hence the ini file.  The ini file is fairly easy to read, note the TMPAR_OBT if it is -1 means to use the time from ‘TM’ lines which is the case with CEA but won’t be with EDDS.   
2.)       Found a way to have the plots with a datetime as the x-axis.
3.)       Finally added a small feature to try and rewind/fast forward in time.  Hence the code stores file pointer locations when it can to help jump back/forward. Likewise you will see a ‘Acquire Full Range’ button to read in the full csv file and accumulate a time range, this is needed because when the HKMonitor starts it will start at the end of the file and people may want to go back further in time so you have to reread earlier points.
Link:
http://msslkz.mssl.ucl.ac.uk/usdownloads/euclid/HKTM_monitor_New.zip
 
How to run:
python hktm_monitor_IWS_CC.py CEA_ORIG
 
I have a .sh script that had these few lines on my desktop:
eval "$(conda shell.bash hook)"
conda activate IWSop
cd ~/HKTM_monitor
export PYTHONPATH=.
python hktm_monitor_IWS_CC.py CEA_ORIG
 
**Start up the basic ‘feed’ script to output to a new file.  Browse to this new file on the HK Monitor and ‘Launch’ then click on Real time mode.
**NOTE: Noticed it was a good minute or two till I saw lines plotted on the screen.  Partly reason is y-value is around 30 but some of the early lines in the CEA file start off at -273.
**NOTE: Be prepared for  a lot of print lines with the word ‘KEVIN’ on it.  Trying to debug what was happening earlier during the ‘ART’ data.  This is coming from EchoReader.py
 
Tasks:
1.)       After some time the Graph will crash.  It is because I was not certain what to do when we fill up all the data points.  If you look at GraphFrame.py you will see a MAXLEN and used in update_plots.  So I think this is telling me that update_plots is redrawing all points you see in the x_axis&y_axis.  What should happen when we go above MAXLEN?   Only thing I can think of is maybe have the arrays shifted in such that last half goes to first half of the array (then last half is blank to fill up again).  What do you think is simple and works well?
(Could increase MAXLEN that means more data points are plotted on each update_plot but eventually run into memory issue and you still have to face the music that your going to go above MAXLEN).
 
2.)       I think it is calling update_plot on every line detected in the csv file.  EchoReader.py.  I have a feeling this is something that I did and was better beforehand where it was not needing to call update_plots every time.  I wonder if you might have a look and see If its ok, feel like I is calling to update the graph more often than it should.
 
3.)       Low priority and something to discuss later: There is a task related about them jumping back in time for let’s say 5min min.  What originally did was clear out the graph and basically reset the livestream to a file pointer in the csv file acting like a live stream again just further back in time.   It worked but realized this is a bit silly making it act like a live stream when there is plenty of data to plot, hence it should just try to read in a lot of files and throw up an entire plot (live stream could be paused till they go to the end of the file again).