Data File Support

All NetCharts applets support the use of the FILE keyword to retrieve parameter definitions or data values from one or more data files. The FILE keyword allows HTML authors and developers to separate the chart display parameters (usually defined in the HTML file) from the actual data values, which may be more easily stored in data files.

The following specific issues are addressed in this document:

Basic Data File Definitions

The FILE keyword can be used to define any parameter, regardless of the number of items or attributes required by that parameter. That is, the FILE keyword can be used to define a single value parameter, such as Bar3DDepth, a single tuple of attributes, like the Grid Parameter, or a vector of individual or tuple values, like the Slices and LineSets parameters. The most common use of the FILE keyword is for defining a vector of data values in a comma delimited file.

The FILE keyword has the following syntax:

Parameter = FILE "url pathname";

The filename given can be any URL that is valid for the environment in which the chart is being executed. For example, in a browser, if the chart is downloaded from a Web server, then the URL can specify a relative pathname or a full HTTP pathname.

For example, the following parameter definitions specify a local file named barlabels.dat and another data file, named barset1.dat, located on a specific server:

BarLabels = FILE "barlabels.dat";

DataSets  = ("My Data");

DataSet1  = FILE "http://mycompany.com/data/barset1.dat";

If a relative pathname is specified, as in the BarLabels parameter above, then the DocumentBase will be used as the start of the relative location.

CGI Script Data Output

Since the FILE keyword allows an arbitrary URL to be specified for the "data file", it is possible to use the output from a CGI script to define a specific parameter.

For example, the following parameter definition specifies a CGI script, with appropriate input parameters, that will generate a text file containing the desired data.

DataSet1 = FILE "/cgi-bin/genData?user=fred&dataset=barset1";

In this case, the genData script is executed, generating whatever data is appropriate for the given CGI input parameters. NetCharts will automatically invoke the script and process all results returned, using them to define the given parameter.

This capability allows Web developers to create scripts that simply generate the necessary data, while leaving all NetCharts syntax in the HTML file(s) that need that data. This eliminates the need to generate the entire HTML file, just to insert a few data values in the chart definition. (Naturally, we don't restrict you from doing that, if you prefer. The FILE keyword is just another alternative.)

FileFormat Directive

Regardless of how the data file is stored, accessed or generated, it is automatically processed by the chart in order to define one or more attributes, for one or more tuples. This processing is controlled by a small number of attributes, defined in the FileFormat directive, that specify the "delimiters" of the data. Each delimiter specifies one or more characters that are used in the file to separate the various data components.

The FileFormat directive has the following syntax:

FileFormat = ("startDelim", "itemDelim", "endDelim",
              "whiteSpace", "quoteChars", "commentChars");

Any number of FileFormat directives can be specified within a parameter script, to control the processing of different data files. The attributes remain in effect for all subsequent FILE keywords, until the next FileFormat directive is defined.

For example, the following script uses two files with the default format, followed by two other files with different startDelim and endDelim values.

BarLabels      = FILE "barlabels.dat";
BarActiveLabel = FILE "baractive.dat";

FileFormat = ("{", null, "}");

DataSet1 = FILE "dataset1.dat";
DataSet2 = FILE "dataset2.dat";

All of the FileFormat attributes are optional and if they are not specified, or they are specified to be "null", then the previously defined value or the default value will be used.

The delimiter strings can contain any number of characters. Each character in the string may be used as a delimiter in the data file. Generally, only one character is used per delimiter, although this is not necessary. The delimiters are used as follows:

Delimiter Default Usage
startDelim "(" Denotes the start of a tuple of attribute values. If any of the characters in this string are found, a new tuple will be started. Start delimiters are optional at the beginning of a tuple.
itemDelim "," Delimits two attributes within a tuple definition.
endDelim ")\n" Denotes the end of a tuple. When seen, the attributes that have been processed since the start of the tuple will be assigned to the current parameter.
whiteSpace " \t" All of the characters specified in this string will be ignored when parsing attribute values. If whitespace is required within a value, the value must be quoted.
quoteChars "\"\'" Any of these characters may be used to denote a value string, that may include whitespace and other delimiters.
commentChars "#" Every character in this string denotes the beginning of a comment in the data stream. All characters following the comment character will be ignored, up until the next line feed.

Of all of the delimiters that can be used, only the itemDelim and endDelim characters must be present within the data stream to define values.

For example, the following data file defines a vector of X/Y data value pairs for a data set in an XY Chart.

#
#   DataSet1.dat
#
10, 20
20, 30
30, 40
#
#   End of File
#

In this case, no startDelim character is used (even though the default specification defines "(" as a startDelim), and the linefeed is used to denote the end of a tuple. That data file could also be written as follows, with the same results.

#
#   DataSet1.dat
#
(10, 20) (20, 30) (30, 40)
#
#   End of File
#

In this case, the "(" and ")" symbols are used to denote the start and end of the tuples. Finally, the following alternative could be used to achieve the same results again.

#
#   DataSet1.dat
#
(10,20)
(20,30)
(30,40)
#
#   End of File
#

This data file actually uses two different symbols to denote the end of each tuple (e.g., ")" and a line feed). NetCharts automatically detects the redundancy and properly generates only three vector items.

Although the default delimiters are consistent with NetCharts parameter definitions and are generally easy to read and generate, developers are free to specify any set of delimiters to suit their data processing needs.

For example, the following FileFormat definition specifies that the given data file will contain a tuple of data on each line (endDelim = "\n"), with a "|" character used to separate tuple attributes. It also specifies the "/" symbol as the beginning of a comment.

FileFormat = ("", "|", "\n", null, null, "/");

Slices     = FILE "pietest.dat";

//
//   pietest.dat
//

10|blue|"Fred Smith"
20|red|"Sally Jane"
30|green|"Oscar Jones"

//
//   End of File
//

Default Attributes

NetCharts parameter definitions allow the user to skip individual attributes within a tuple or specify only the first few. Any un-defined attributes are assigned default values, based on the system-defined defaults or the attributes defined in the previous tuple in a vector.

This feature is also supported when processing data files. Specifically, if a given attribute is not defined, then the default value will be used. This feature is different from the NULL value feature, which allows a user to specify a NULL value for a given attribute. In that case, the NULL is interpreted according to the type of parameter being processed. (See Parameter Definitions for a detailed discussion.

In the following example, the second attribute in each tuple is skipped, which results, in this case, in a default color being assigned to each pie slice.

#
#   pietest.dat
#
(10,,"Fred Smith")
(20,,"Sally Jane")
(30,,"Oscar Jones")
#
#   End of File
#

Please note that the Slices tuple accepts more than 3 attributes, so since the remaining attributes in each tuple are un-defined, the default values are used.

Dynamically Updated Charts

The FILE support in NetCharts is 100 percent compatible with the dynamic update feature provided through the NFParamServer parameter. In that case, a Parameter Server is used to dynamically generate NetCharts parameters, interspersed with "Update" commands to update the chart being displayed. Since FILE support is provided through the same parameter syntax, it can also be generated dynamically by the Parameter Server.

This integration of features allows developers to either extract the data directly from a file in the Parameter Server and then forward it to the chart, or to simply instruct the chart to pull the data directly from the file when it needs to be updated. In both cases, the Parameter Server is in complete control of what is updated and when those updates occur.

See Parameter Server for more details concerning the use of a Parameter Server for dynamic chart updates.

Embedded Applets/Applications

One of the easiest methods of controlling a NetCharts applet from within another Java applet or application is through the use of the loadParams() method. (See Parameter for details.) That method accepts any arbitrary String expression that is parsed to determine the desired parameter definitions. Since the FILE keyword is implemented using the same parameter definitions, it can also be used in conjunction with the loadParams() method for embedded charts.

Trouble-Shooting

If you have trouble with file processing using the FILE keyword, you can use the DebugSet and DebugClear directives to generate debug information on the Java Console for FILE operations, using the following syntax:

DebugSet = FILE;

DebugClear = FILE;

The DebugSet command enables the printing of debug messages for all subsequent FILE keywords, including the processing of all FileFormat Parameters. The debug messages will continue to be printed until the end of the Parameter script or the DebugClear directive is seen.

For example, the following script defines a simple piechart, enabling debug messages for all FILE statements:

DebugSet   = FILE;

FileFormat = ("", "|", "\n", null, null, "/");

Slices     = FILE "pietest.dat";

Resulting in the following debug output on the Java Console:

NFFile: startDelim = <>
NFFile: itemDelim = <|>
NFFile: endDelim = <\n>
NFFile: comments = </>
NFFile: +++++++++++++++++++++++
NFFile: Parameter = Slices
NFFile: Filename = <pietest.dat>
NFFile: Opening http://mycompany.com/reports/pietest.dat
NFFile: Item(1): 10|blue|Fred Smith
NFFile: Item(2): 20|red|Sally Jane
NFFile: Item(3): 30|green|Oscar Jones
NFFile: Processed 3 Item(s)
NFFile: -----------------------