Live File Processor Agent Example – YSI-6

<< Click to Display Table of Contents >>

Navigation:  Live > Agents > File Agents > File Processor > Examples >

Live File Processor Agent Example – YSI-6

1.Add the following settings to the ST_CONFIG table (CONFIG_SECTION = 'Live.FileProcessor', CONFIG_KEY = 'Agent').

 

OBJECT_TYPE

OBJECT_VALUE

STRING_VALUE

Description

YSI-6

dateTimeColumnHeaderRegex

Date|Date Time

DT_LOGGER_DATUM.DATUM_UTC_DT date; column header text contains Date, or Date Time.

YSI-6

dateTimeFormats

M/d/y|d-M-y H:m

Note: This is not a realistic example. The dates will probably be either month-first or day-first, not both. Also, this may be problematic if the date and time can be interpreted as both d/M/y and M/d/y.

YSI-6

folderWatcherPath

C:\EarthSoft\Live\YSI-6


YSI-6

headerLinesRegex

Date.*Time\n.*

The header line contains Date + anything (.*) + Time.

YSI-6

loggerCodeFileNameRegex
Groups

^(?'a'[^_.]*)

DT_LOGGER.LOGGER_CODE = file name text before first underscore (_) or dot (.).

YSI-6

seriesNameColumnHeaderRegex
Groups

^(?'a'.*)\n

DT_LOGGER_SERIES.SERIES_NAME

YSI-6

seriesUnitColumnHeaderRegex
Groups

\n(?'a'.*)$

DT_LOGGER_SERIES.SERIES_UNIT

YSI-6

timeColumnHeaderRegex

Time

DT_LOGGER_DATUM.DATUM_UTC_DT time; column header text contains Time.

YSI-6

utcOffsetDefaultValue

-5

DT_LOGGER_DATUM.DATUM_UTC_DT = file dates and times + 5 hours (Eastern Standard Time; UTC – 5 hours; not adjusted for DST).

 

 

Example File 1

 

2.Start the agent and copy the following file, named SpringField_A_2019-02-01.csv, to folder C:\EarthSoft\Live\YSI-6.

 

 

"Date","Time","Temp","SpCond","Depth","pH","pHmV","Turbidity+","ODO%","ODO Conc"

"M/D/Y","hh:mm:ss","C","mS/cm","m","","mV","NTU","%","mg/L"

"10/11/11","12:21:06",24.32,0.399,0.362,7.59,-55.2,0.0,92.5,7.73

"10/11/11","12:21:43",24.36,0.400,1.112,7.59,-55.1,-0.1,92.3,7.71

"10/11/11","12:22:14",24.37,0.400,2.028,7.59,-54.8,0.0,92.1,7.70

 

 

3.The file's name contains the DT_LOGGER.LOGGER_CODE field. (The location/logger code is not in a YSI 6-series file's header or data lines; it can only be obtained from the file's name, or a default value.) Find/add the logger row. For example:

 

LOGGER_CODE

LIVE_DATA_SOURCE

UTC_OFFSET_HRS

SpringField

folder://COMPUTER_NAME/C:/EarthSoft/Live/YSI-6

-5 (= utcOffsetDefaultValue)

 

4.The file's Date,Time line, and the line that follows it, are the header lines; they contain the DT_LOGGER_DATUM.DATUM_UTC_DT columns Date and Time. The other columns are the series; they contain DT_LOGGER_SERIES.SERIES_NAMEs and SERIES_UNITs.

 

5.For each subsequent line:

a.For each series (each column, excluding Date and Time columns):

i.Find/add the series row for the SERIES_NAME.

ii.Add the datum row (e.g., for the first three series in the first data line):

 

SERIES_NAME

DATUM_UTC_DT

DATUM_VALUE

SERIES_UNIT

Temp

2011-10-11 17:21:06 (= 2011-10-11 12:21:06 - utcOffsetDefaultValue)

24.32

C

SpCond

2011-10-11 17:21:06

0.399

mS/cm

Depth

2011-10-11 17:21:06

0.362

m

 

 

Example File 2

 

The following file has one date and time column, instead of two.

 

 

Date Time,Temp,SpCond,DOsat,DO,Depth,pH,pH,Baro,Battery,TDS

D-M-Y HH:MM:SS,C,uS,%,mg/L,meters,,mV,mmHg,volts,g/L

26-07-13 14:18,19.36,492,108.6,9.99,0.082,7.71,-89.6,768.7,11.9,0.32

26-07-13 14:18,19.36,492,109.6,10.08,0.093,7.71,-89.5,768.6,11.9,0.32

26-07-13 14:18,19.37,492,110.5,10.16,0.145,7.71,-89.6,768.7,11.9,0.32

 

 

 

Note: The date and time formats in the file's header, line:
 
"M/D/Y","hh:mm:ss","C","mS/cm","m","","mV","NTU","%","mg/L"
 
are not used/supported. Instead, the dateTimeFormats are used.
 
Also, in Example File 2, the file's header includes seconds, but the data lines do not have seconds.