This appendix describes how to use the GoldenGate data conversion services. It covers these topics:
“Converting Data Using the GoldenGate Data Conversion Service” explains how to use the converters provided.
“Compiling and Linking Your Program with GoldenGate” describes the header file to use when compiling and linking a program.
“Writing Converters for the GoldenGate Data Conversion Service” explains how to customize your own converters.
This section describes how you can use the GoldenGate data conversion service in your application. Specifically, it explains:
“Overview of the Conversion Process” describes the steps involved in converting data using GoldenGate.
“Selecting a Converter ” describes how to select a converter by querying the converter registry and setting up the conversion context.
“Using GoldenGate to Convert Data” describes how to initialize the conversion pipeline, send data through it, and clean up after the conversion.
To convert data using GoldenGate, follow these steps:
Choose a converter.
Obtain a list of converters that read the source format and write the target format.
Create a conversion context structure and set conversion parameters.
Evaluate the list of converters to determine which one is best suited for the current conversion.
Convert data.
There are two methods of converting data, depending on whether the data is in a stream or in a file.
Converting Stream Data
Initialize the selected converter.
Send data through the converter and read results back.
Clean up resources by destroying the conversion context.
Converting Data Files
Call the file conversion function.
Clean up resources by destroying the conversion context.
GoldenGate maintains a list of available converters in the converter registry. This registry contains an entry for each converter, specifying characteristics such as the type of input data it takes and the type of output data it produces. To find out if there are any converters that will convert from format “A” to format “B,” you can query the registry.
GoldenGate returns a list of converters that take the specified input and produce the specified output. You can be as specific as you like when querying the registry, to ensure that only relevant converters are listed. You should also use the query to eliminate inappropriate categories of converter, such as those of type StreamToStream if you are converting a file. If the list contains more than one converter, you may need to evaluate the converters to see which one best meets your needs. Even if the list contains only one converter, you should evaluate it to make sure it can handle your conversion request.
To communicate with a converter, you must create a conversion context. The conversion context is a data object that stores conversion parameters. The conversion context is passed to subsequent library calls that set input and output parameters, evaluate converters, initialize the conversion pipeline, and move data through it.
Once you have created a conversion context and specified the desired conversion parameters, you can evaluate the list of converters you obtained when you queried the registry. For example, suppose you want to convert from one audio format to another and change the sample rate at the same time. Querying the registry returns a list of converters that will convert between the specified input and output formats. To determine if any of these converters will perform the desired sample rate conversion, you have to create a conversion context, set the desired parameters (including input and output sample rate) and then evaluate the individual converters.
It's best to evaluate a converter before you invoke it to perform a conversion. You do this for the following reasons:
Evaluation gives the converter an opportunity to inspect your data parameters. Some converters will have more functionality than others, even though their input and output types are the same. A well-designed converter will know just by looking at parameters whether it can do the conversion.
Conversion is typically an expensive operation. If your attempt to convert fails, you can still choose a different converter and try again, but you could have avoided lost time by trying a converter that can accept your specific request.
Depending on your needs, you can select the first converter on the list that passes the evaluation stage, or evaluate the whole list and use your own rules to choose between those that pass.
Once you determine the converter to use, the final stage depends on whether you are converting data in a file or a stream.
If you are converting a stream, initialize a conversion pipeline that reads your stream and passes back results as they are available. Then you send all your data through the pipeline and read the results until you see the end of stream marker for the pipeline. At this point, terminate the pipeline. This causes GoldenGate to clean up data structures it keeps for maintaining a stream conversion.
If you are converting a file, the procedure is simpler. You call a single GoldenGate function to perform the operation, and wait for results. If necessary you can provide a callback function that will notify you when results become available. This allows you to service other events going on in your application during what may be a long conversion.
This section describes how to select a converter by querying the converter registry and setting up the conversion context. Specifically, this section covers:
“Querying the Converter Registry”, which explains how to obtain a list of possible converters.
“Setting Up the Conversion Context”, which describes how to create a conversion context.
“Evaluating Converters”, which explains how to find a converter that performs the specified conversion.
“Getting Converter Details”, which describes how to get a description of a converter.
“Converter Return Status Values ”, which lists return status values.
To query the converter registry, you specify a set of constraints. Each constraint consists of an attribute (such as input format), a value for the attribute, and a comparison operator. For example, you can ask for a converter that has input format equal to “AIFF_FILE,” and version number greater than 2. Use the SgCvtSetQueryConstraint() function to fill in an array of SgCvtQueryConstraint structures, then pass the array to the SgCvtQueryRegistry() function. The following code fragment demonstrates a simple query that locates converters capable of converting AIFF_FILE to WAVE_FILE:
SgCvtQueryConstraint constraints[2]; SgCvtStatus status; SgCvtConverterId *converters; int num_constraints, num_converters; SgCvtRegistry registry = NULL; status = SgCvtSetQueryConstraint(constraints[0], SG_CVT_ATTR_INPUT_TYPE, "AIFF_FILE", SG_CVT_OP_EQ); status = SgCvtSetQueryConstraint(constraints[1], SG_CVT_ATTR_OUTPUT_TYPE, "WAVE_FILE", SG_CVT_OP_EQ); num_constraints = 2; status = SgCvtQueryRegistry(constraints, num_constraints, ®istry, &converters, &num_converters); |
The SgCvtQueryRegistry() function returns an array of converter IDs that can be used to identify the individual converters.
The registry argument specifies the GoldenGate converter registry to be queried. During this call, the registry is located on disk (/var/GoldenGate/ConverterRegistry by default), and its contents parsed to find a converter that matches your requirements.
The first time you call SgCvtQueryRegistry, specify registry as NULL as in the previous example, which causes this lookup. When you have finished converting, you can either call SgCvtFreeRegistry to release the resources that GoldenGate may have cached after reading the file, or you can re-use the value returned in registry for subsequent queries, avoiding the overhead of looking up the file.
If you choose to free the registry between queries, your program will always have the latest information, even if the registry changes while your program is running. If you choose to re-use the registry handle, you have no control over whether or not GoldenGate will re-parse the registry. It will try to use its cache first. If for any reason the cache is invalid, GoldenGate may at its discretion rebuild it by reading the disk-based registry again.
The converters argument returns an array of matching converter IDs, of which the first num_converters are valid and matched the query. You should free this array when you are finished using it, using free(3).
Table G-1 lists the attributes you can query.
Table G-1. Converter Attributes
Attribute Name | Description |
---|---|
SG_CVT_ATTR_NAME | Converter name |
SG_CVT_ATTR_INPUT_FORMAT | Input format |
SG_CVT_ATTR_OUTPUT_FORMAT | Output format |
SG_CVT_ATTR_IO_METHOD | StreamToStream or FileToFile |
SG_CVT_ATTR_INPUT_LABEL | Input format, human readable version |
SG_CVT_ATTR_OUTPUT_LABEL | Output format, human readable version |
SG_CVT_ATTR_VENDOR | Vendor's name |
SG_CVT_ATTR_VERSION | Vendor's version information |
SG_CVT_ATTR_DESCRIPTION | Description of converter |
Most of the time, you'll be interested in the input format and output format attributes. “Supported Target Formats” in Chapter 7 lists common data formats. Other attributes may be useful when listing converters for users. For example, if you want the user to choose between two converters that perform the same conversion, you can display the vendor names and version numbers.
Table G-2 lists the operators you can use in your query.
Operator | Symbol |
---|---|
equal to | SG_CVT_OP_EQ |
not equal to | SG_CVT_OP_NE |
less than | SG_CVT_OP_LT |
less than or equal to | SG_CVT_OP_LE |
greater than | SG_CVT_OP_GT |
greater than or equal to | SG_CVT_OP_GE |
Note that if more than one constraint is specified on a single attribute, a logical AND is implied. For example, you can select a range of version numbers by setting “version greater than or equal to one” as one constraint and “version less than or equal to three” as a second constraint.
SgCvtQueryConstraint can return the following status value:
SG_CVT_E_SUCCESS | |
The operation succeeded. |
SgCvtQueryRegistry can return the following status values:
SG_CVT_E_SUCCESS | |
The operation succeeded. | |
SG_CVT_E_FAILURE | |
Could not find the registry, or failed to parse it. Most likely when the default registry has been edited to add new converters, and a syntax error introduced. You may also be loading the wrong file. Make sure that if there is a file called ConverterRegistry on your path, it is a valid registry using the CDF syntax. Also make sure the CVT_REGISTRY_OVERRIDE variable is not set. |
Before you can evaluate or use a converter, you must create a conversion context and set parameters governing the conversion. Use the SgCvtCreateConversionContext() function to create a conversion context:
SgCvtStatus SgCvtCreateConversionContext(SgCvtConversionContext *context) |
SgCvtCreateConversionContext can return the following status values:
SG_CVT_E_SUCCESS | |
The operation succeeded. | |
SG_CVT_E_NOMEM | |
Insufficient memory to allocate a context. |
Next, set any digital media parameters that affect your conversion by calling SgCvtSetContextInfo.
SgCvtStatus SgCvtSetContextInfo ( SgCvtConversionContext context, unsigned long valuemask, SgCvtContextInfo *context_data ); |
where
context | specifies the context you created with SgCvtCreateConversionContext | ||
valuemask | specifies which fields in the SgCvtContextInfo structure are being set in the context. This is specified as any of the following OR'ed together:
| ||
context_data | specifies the values being set |
SgCvtSetContextInfo can return the following status value:
SG_CVT_E_SUCCESS | |
The operation succeeded. |
See the IRIS Media Libraries Programming Guide for information on setting DMparams.
To evaluate a converter, call SgCvtEvaluateConverter():
SgCvtStatus SgCvtEvaluateConverter(SgCvtConverterId converter_id, SgCvtConversionContext context DMparams **output_params) |
where
converter_id | is a converter ID returned by the SgCvtQueryRegistry() function | |
context | is a valid conversion context obtained from SgCvtCreateConversionContext() | |
output_params | returns the output of the request. Converters may set these parameters, even though they accept the request. |
SgCvtEvaluateConverter() can return the following status values:
SG_CVT_E_ACCEPT | |
The converter can perform the conversion specified by the conversion context. | |
SG_CVT_E_REJECT | |
The converter can't perform the requested conversion. |
When evaluating a converter returns a status of SG_CVT_E_ACCEPT, you should take one final step before calling the converter. You should inspect the output_params argument, which returns a DMparams list describing the result that the converter will produce. If your program has very strict requirements, this will help protect you if the converter has accepted the request but cannot honor what it considers a minor parameter, or if you passed a parameter it could not understand.
If your program needs to display information about available converters, or do other processing based on the data stored about a converter in the converter registry, call SgCvtGetConverterAttributes() to get a description of it. The function prototype for SgCvtGetConverterAttributes() is shown below.
SgCvtStatus SgCvtGetConverterAttributes(SgCvtConverterId converter_id, unsigned long converter_attr_mask, SgCvtConverterAttrs *attributes) |
When you are finished using the fields of the SgCvtConverterAttrs structure, you should free the string attributes and the structure itself (if you allocated it dynamically) using free(3C).
SgCvtGetConverterAttributes can return the following status value:
SG_CVT_E_SUCCESS | |
The operation succeeded. |
This section describes the different methods you can use to convert data. Topics include:
Your file-based data is always converted using the function SgCvtConvertFileToFile. Before you call it however, you need to decide whether you want the function to block while the conversion is going on, or return immediately and let you know later that the conversion is complete.
In many cases blocking mode is sufficient, and it is much simpler to use if your program is not naturally event driven. However, if your application has a GUI, you may prefer non-blocking mode because it allows your event loop to keep running while conversion is going on. When conversion is complete, you are notified through a callback function that you supply, and you can use the converted data.
Both modes are invoked using SgCvtConvertFileToFile:
typedef void (*SgCvtCallback)(SgCvtConversionContext context, void *client_data, void *callback_data); SgCvtStatus SgCvtConvertFileToFile ( SgCvtConversionContext context, SgCvtConverterId converter_id, char *input_file, char *output_file, unsigned long callback_mask, SgCvtCallback callback, void *client_data ); |
where
context | the conversion context, holding the I/O filenames and parameters | ||
converter_id | the converter ID, returned by SgCvtQueryConverter | ||
input_file | pathname of input file. You must have read permission. | ||
output_file | pathname of output file. You must have write permission. | ||
callback_mask | mask indicating when callback should be called. It should be some logical combination of the following values:
| ||
callback | specifies the callback function | ||
client_data | a pointer to application-defined data structure that will be passed to the callback when invoked |
If specified, the callback argument is the address of the function to call when conditions specified by the callback_mask arise. If the callback function is not specified, or the mask is zero, the function executes in blocking mode.
SgCvtConvertFileToFile can return the following status values:
SG_CVT_E_SUCCESS | |
The operation succeeded. | |
SG_CVT_E_BAD_CONVERTER_TYPE | |
The converter was not registered as FileToFile IO method. | |
SG_CVT_E_READ_FAILED | |
The input file could not be read. It may be missing, or the permissions are insufficient for reading. | |
SG_CVT_E_WRITE_FAILED | |
The output file could not be written. This can happen if the user does not have write permission for the target directory, or if the supplied pathname was invalid. |
To convert data using your specified converter, you must initialize the conversion pipeline, and then send the data through. After reading the last block of converted data, clean up by destroying the conversion context to free the resources associated with the pipeline. This section covers the following topics:
Prepare the converter to receive data by calling SgCvtInitializePipeline():
SgCvtStatus SgCvtInitializePipeline(SgCvtConversionContext context, SgCvtConverterId converter_id) |
where
context | is a valid conversion context obtained from SgCvtCreateConversionContext() | |
converter_id | is a converter ID returned by the SgCvtQueryRegistry() function |
SgCvtInitializePipeline can return the following status values:
SG_CVT_E_SUCCESS | |
The operation succeeded. | |
SG_CVT_E_FAILURE | |
The context or its contents is bad or one of the subprocesses required to host a converter function could not be launched. | |
SG_CVT_E_BAD_CONVERTER_TYPE | |
The converter was not registered as StreamToStream. Converters that are designed to work with streaming data advertise themselves as using the StreamToStream method of I/O in the registry. |
You may send and receive arbitrarily sized blocks of data, so use a block size that is convenient.
Send data to the converter using SgCvtSendData(). The function prototype for SgCvtSendData() is shown below:
SgCvtStatus SgCvtSendData( SgCvtConversionContext context, void *data, size_t length, DMparams *params, boolean_t canwait ) |
where
context | is a valid conversion context | |
data | is a pointer to the data block to be converted | |
length | is the length of the data block | |
params | is a DMparams structure describing the data to be converted | |
canwait | is a boolean value that indicates what the function should do if it cannot send the data immediately. If you specify B_TRUE, SgCvtSendData() will block until it can send the data to the conversion pipeline. If you specify B_FALSE, SgCvtGetData() will return immediately with a status of SG_CVT_E_AGAIN. This status indicates that you should try again. |
SgCvtSendData can return with the following status values:
SG_CVT_E_SUCCESS | |
The operation succeeded. | |
SG_CVT_E_FAILURE | |
An I/O error occurred while trying to send data through the pipe connecting two pipeline components. | |
SG_CVT_E_AGAIN | |
Required resources were temporarily unavailable. The caller should retry later. |
Read data from the converter using SgCvtGetData(). The function prototype for SgCvtGetData() is shown below:
SgCvtStatus SgCvtGetData( SgCvtConversionContext context, size_t buf_len, void *buffer, size_t *length_returned, DMparams **params_returned, boolean_t canwait ) |
where
context | is a valid conversion context | |
buf_len | specifies the size of buffer | |
buffer | is a pointer to a pre-allocated buffer of at least buf_len bytes. | |
length_returned | is the actual length of the returned data (this may be less than bytes_requested if non-blocking mode is specified, or if the converter encounters end-of-stream) SgCvtGetData | |
params_returned | is a DMparams structure describing the converted data. | |
canwait | is a boolean value that indicates what the function should do if no data is available. If you specify B_TRUE, SgCvtGetData() will block until data becomes available from the conversion pipeline. If you specify B_FALSE, SgCvtGetData() will return immediately with a status of SG_CVT_E_QUEUE_EMPTY. This status indicates that you should try again. |
SgCvtGetData can return the following status values:
SG_CVT_E_SUCCESS | |
The operation succeeded. | |
SG_CVT_E_FAILURE | |
An I/O error occurred while trying to read data from the pipe connecting two pipeline components. | |
SG_CVT_E_AGAIN | |
Required resources were temporarily unavailable. The caller should retry later. | |
SG_CVT_E_END_OF_STREAM | |
The operation succeeded, and the end of the data has been reached. |
The non-blocking mode of SgCvtSendData() and SgCvtGetData() allows programs to continue working on other tasks (such as handling events from a graphical interface) while waiting to send data to or read data from the conversion pipeline.
When you've sent the last of the data to the converter, call SgCvtSendEndOfStream() to indicate the end of the data. After you've read the last of the converted data, free the resources associated with the conversion context by calling SgCvtDestroyConversionContext():
SgCvtStatus SgCvtSendEndOfStream( SgCvtConversionContext context ); SgCvtStatus SgCvtDestroyConversionContext( SgCvtConversionContext context ) |
If you need to terminate the conversion process before reaching the end of the data, call SgCvtDestroyConversionContext().
SgCvtSendEndOfStream can return the following status value:
SG_CVT_E_SUCCESS | |
The operation succeeded. |
SgCvtDestroyConversionContext can return the following status value:
SG_CVT_E_SUCCESS | |
The operation succeeded. |
Table G-3 lists converter functions and their return status values.
Table G-3. Converter Return Status Values
Function | Return Value | Description |
---|---|---|
SgCvtSetQueryConstraint | SG_CVT_E_SUCCESS | The operation succeeded. |
SgCvtQueryRegistry | SG_CVT_E_SUCCESS | The operation succeeded. |
| SG_CVT_E_FAILURE | Could not find the registry, or failed to parse it. Most likely when the default registry has been edited to add new converters, and a syntax error introduced. You may also be loading the wrong file. Make sure that if there is a file called ConverterRegistry on your path, it is a valid registry using the converter description file syntax. Also make sure the CVT_REGISTRY_OVERRIDE variable is not set. |
SgCvtGetConverterAttributes | SG_CVT_E_SUCCESS | The operation succeeded. |
SgCvtCreateConversionContext | SG_CVT_E_SUCCESS | The operation succeeded. |
| SG_CVT_E_NOMEM | Insufficient memory to allocate a context. |
SgCvtDestroyConversionContext | SG_CVT_E_SUCCESS | The operation succeeded. |
SgCvtSetContextInfo | SG_CVT_E_SUCCESS | The operation succeeded. |
SgCvtGetContextInfo | SG_CVT_E_SUCCESS | The operation succeeded. |
SgCvtEvaluateConverter | SG_CVT_E_ACCEPT | The converter can perform the requested conversion. |
| SG_CVT_E_REJECT | The converter cannot perform the requested conversion. |
SgCvtInitializePipeline | SG_CVT_E_SUCCESS | The operation succeeded. |
| SG_CVT_E_FAILURE | The context or its contents is bad or one of the subprocesses required to host a converter function could not be launched. |
| SG_CVT_E_BAD_ | The converter was not registered as StreamToStream. Converters that are designed to work with streaming data advertise themselves as using the StreamToStream method of I/O in the registry. |
SgCvtTerminatePipeline | SG_CVT_E_SUCCESS | The operation succeeded. |
SgCvtSendData | SG_CVT_E_SUCCESS | The operation succeeded. |
| SG_CVT_E_FAILURE | An I/O error occurred while trying to send data through the pipe connecting two pipeline components. |
| SG_CVT_E_AGAIN | The required resources were temporarily unavailable. The caller should retry later. |
SgCvtGetData | SG_CVT_E_SUCCESS | The operation succeeded. |
| SG_CVT_E_FAILURE | An I/O error occurred while trying to read data from the pipe connecting two pipeline components. |
| SG_CVT_E_AGAIN | The required resources were temporarily unavailable. The caller should retry later. |
| SG_CVT_E_END_OF_ | The operation succeeded, and the end of the data has been reached. |
SgCvtSendEndOfStream | SG_CVT_E_SUCCESS | The operation succeeded. |
SgCvtEncodeParams | SG_CVT_E_SUCCESS | The operation succeeded. |
SgCvtDecodeParams | SG_CVT_E_SUCCESS | The operation succeeded. |
| SG_CVT_E_NOMEM | Insufficient memory to allocate structures. |
SgCvtFreeEncodedParams | SG_CVT_E_SUCCESS | The operation succeeded. |
| SG_CVT_E_FAILURE | The data could not be decoded. |
SgCvtConvertFileToFile | SG_CVT_E_SUCCESS | The operation succeeded. |
| SG_CVT_E_BAD_ | The converter was not registered as FileToFile I/O method. |
| SG_CVT_E_READ_ | The input file could not be read. It may be missing, or the permissions are insufficient for reading. |
| SG_CVT_E_WRITE_ | The output file could not be written. This can happen if the user does not have write permission for the target directory, or if the supplied pathname was invalid. |
SgCvtGetFileSelectionTarget | SG_CVT_E_SUCCESS | The operation succeeded. |
| SG_CVT_E_UNKNOWN
_ | The file type could not be determined. |
| SG_CVT_E_NO_TARGE T | The selection target for the type of file could not be determined, or there is none. |
| SG_CVT_E_FAILURE | The operation could not be performed for another reason, such as the underlying file typing database library could not be accessed, or the database itself was corrupt or missing. |
SgCvtIsPipeline | B_TRUE | The translator is a multi-stage pipeline. |
| B_FALSE | The translator is a single-stage converter. |
To compile and link your program, you need to include the header file SgCvt.h and include the library libcvt in your link line.
An example of a simple GoldenGate program follows. It includes the required header file, enumerates the registered converters, and prints their input and output labels.
#include <SgCvt.h> main(int argc, char **argv) { SgCvtRegistry registry = NULL; int n=0; SgCvtStatus s; SgCvtConverterId *cvtrs; int ncvtrs; s = SgCvtQueryRegistry(NULL, 0, ®istry, &cvtrs, &ncvtrs); for (n=0; n<ncvtrs; n++) { SgCvtConverterAttrs attrs; SgCvtGetConverterAttributes(cvtrs[n], SG_CVT_ATTR_FLAG_INPUT_LABEL | SG_CVT_ATTR_FLAG_OUTPUT_LABEL, &attrs); printf(“%d %25s -> %s\n”, n+1, attrs.input_label, attrs.output_label); free(attrs.input_label); free(attrs.output_label); } SgCvtFreeRegistry(registry); } |
The following Makefile illustrates the compile and link requirements for this program.
# # Makefile for GoldenGate Listing sample program # CC = cc TARGET = gg_listing SOURCES = $(TARGET). INCLUDES= -I/usr/include/convert REQLIBS = -lcvt all: $(CC) -o $(TARGET) $(INCLUDES) $(SOURCES) $(REQLIBS) |
This section describes how to write converters that can integrate with GoldenGate and become available to any component that is aware of GoldenGate. The following information assumes that you are familiar with the interfaces described in “Converting Data Using the GoldenGate Data Conversion Service”. Both converters and applications use many of the functions and data structures.
Creating a GoldenGate data converter involves writing the converter and building the DSO, then testing, registering, and installing the converter. The topics below describe:
“Writing Converter Code” explains how to write the code that converts the data, or choose an existing command that you want to make available through the conversion service.
“Building a DSO” describes how to create a Dynamic Shared Object (DSO) and write a registry entry using converter description file syntax.
“Testing Your Converter” explains how to test your converter.
“Registering Your Converter” describes how to register your converter to make it available to GoldenGate clients.
“Installing Your Converter ” lists the library location for converter DSOs.
“Some Sample Converters” shows annotated code for two converters.
This section describes how to write converter code and includes the following topics:
When the operation field of the SgCvtConverterData structure passed to your converter is equal to SG_CVT_REQ_EVALUATE, your converter should inspect the input, output, and meta parameters held in the conversion context and determine whether or not it can satisfy the request, without actually performing conversion.
If your converter can satisfy the request, it should set the status_return field of the SgCvtConverterData structure to SG_CVT_E_ACCEPT before returning. Otherwise it should set status_return to SG_CVT_E_REJECT.
When the operation field of the SgCvtConverterData structure is equal to SG_CVT_REQ_CONVERT, your converter must extract the necessary information from the SgCvtConverterData structure it is passed, and perform the conversion if possible. If conversion is successful, it should return with the status_return field set to SG_CVT_E_SUCCESS, and if it is unsuccessful, the status_return field should be set to either SG_CVT_E_FAILURE or a more specific error code if appropriate (see the error codes available in SgCvt.h).
How the converter is implemented depends on whether you are writing the conversion code yourself, or simply using an existing command-line converter.
If you are creating a “wrapper” to make an existing UNIX command available through the GoldenGate conversion service, the procedure is quite straightforward.
In this case, your converter is a function that gathers the input and output requirements from its arguments, and executes the external UNIX command (for instance, by calling the system(2) function).
Your function should do as much checking as possible to ensure that the external command can work. For instance, you should verify that the command is installed before calling it, and that you have execute permission.
Also check for appropriate permissions to read input files and write output files, in the case of file converters. If you detect an error before calling the command, return an error status in the status field of the data argument.
For example, the code below shows a FileToFile converter that wraps an existing UNIX command rtf2html. You will find other fully annotated examples at the end of this section.
#include <libgen.h> #include <SgCvt.h> void RtfToHtml(void *arg) { SgCvtConverterData *data = (SgCvtConverterData *) arg; SgCvtContextInfo ctx_info; char cmd[BUFSIZ]; int sys_status = 0; char *xlator_path; /* Evaluation - just accept for this example */ if (data->operation == SG_CVT_REQ_EVALUATE) { data->status_return = SG_CVT_E_ACCEPT; return; } /* Conversion */ /* depends on `rtf2html' command being available */ xlator_path = pathfind(getenv (“PATH”), “rtf2html”, “rx”); if (xlator_path == NULL) { data->status_return = SG_CVT_E_MISSING_COMMAND; return; } (void) SgCvtGetContextInfo(data->context, SG_CVT_INFO_INPUT_FILE | SG_CVT_INFO_OUTPUT_FILE, &ctx_info); /* cmd syntax is `rtf2html inputfile outputfile' sprintf(cmd, “%s %s %s 2> /dev/null”, xlator_path, ctx_info.input_file, ctx_info.output_file); sys_status = system(cmd); data->status_return = sys_status ? SG_CVT_E_FAILURE : SG_CVT_E_SUCCESS; return; } |
Notice that GoldenGate passes the necessary information to a converter by reference. The SgCvtConverterData structure is the mechanism for this. It is defined as follows:
typedef struct { SgCvtRequestType operation; SgCvtConversionContext context; DMparams *output_params; SgCvtStatus status_return; } SgCvtConverterData; |
If your converter does not use an external command to translate the data, but does the conversion itself, the structure of the converter function is essentially the same.
You still use the SgCvtConverterData structure to communicate with GoldenGate. Between extracting the necessary arguments from the structure and returning from the function, you just call your own functions that do the conversion.
Your converter should use standardized names for its input and output types wherever possible. This is important because applications are written to request data by a particular name. If your converter uses a different name for the same data format, GoldenGate will not find your converter and the conversion may fail.
See “Supported Target Formats” in Chapter 7 for the data formats supported by the default Silicon Graphics converters.
You can also use your own data format names. However, the name your application uses must match the name you registered so GoldenGate ca l find the converter. However, if you use your own data format names, it is unlikely that other applications will be able to take advantage of your converter. Do this only if the format name is well understood among all the applications you intend to cooperate with.
You can use SgCvtGetData() and SgCvtSendData() in either blocking or non-blocking mode, depending on your requirements. Both modes are described in “Converting Data Using File Converters ”.
If your converter needs to return immediately to do other work, such as tracking activity on an I/O device, you should set the canwait argument to these functions to B_FALSE. If the conversion pipeline is not ready for an immediate read or write operation, the call will return immediately with a status value indicating that nothing happened and that you should try the same operation again. For additional information on the canwait argument, see “Sending and Receiving Data”.
If SgCvtSendData() cannot send data immediately and canwait is B_FALSE, it returns SG_CVT_E_AGAIN. This indicates that your data has not been sent, and you should try the operation again, using the same data.
SgCvtGetData() returns SG_CVT_E_QUEUE_EMPTY if there is no data immediately available and canwait is B_FALSE. You should try the operation again later.
If your converter has no other I/O requirements, you can simplify your code slightly by setting the canwait argument to B_TRUE. You should use this option by default, because it can eliminate redundant context switching to your idling converter, and improve system performance.
There are two categories of converter: FileToFile and StreamToStream.
A FileToFile converter uses the input and output file attributes of the conversion context to get its input and save its output, as shown in the example above.
A StreamToStream converter follows this general procedure after extracting the required parameters from the context:
Fetch a block of input data using SgCvtGetData
Convert the data to the new format
Send converted data back to GoldenGate
The converter repeats these steps until it receives a status of SG_CVT_E_END_OF_STREAM from SgCvtGetData, and it successfully sends all the converted data. Then it calls SgCvtSendEndOfStream to tell GoldenGate it is finished converting, and finally it returns.
The functions used for stream conversion are the same ones used by applications to work with conversion streams:
To fetch input and output parameters to be used in the conversion, use SgCvtGetContextInfo.
To fetch a block on data for conversion, use SgCvtGetData.
To send a block of converted data back to GOldenGate, use SgCvtSendData.
To break your connection to the stream and tell GoldenGate your converter is finished, use SgCvtSendEndOfStream.
Keep in mind the following constraints when writing converters:
You must not use libraries that are unsafe for threads. For instance, you should not use Motif or other GUI libraries that are not “multi-thread-safe.”
You should be careful if installing global event handlers, such as timers and signal handlers, if they override those that may already be installed by the host application. The safest policy is to avoid this altogether.
Where possible, you should avoid intentionally locking system resources such as physical memory blocks by using low-level UNIX calls or device drivers, because this can result in deadlock.
Your code should be reentrant. This means it should not rely on global state between calls, because it is possible for more than one instance of your converter to be running at the same time.
Converters are free to choose the size of the data blocks they read and write. GoldenGate writes into the buffer that your converter supplies during a SgCvtGetData() call. Your converter must allocate and free this buffer space as necessary. During a SgCvtSendData() call, your converter again supplies a buffer of data. The SgCvtSendData() call does not alter your buffer. If the call returns SG_CVT_E_SUCCESS to indicate that your data has been sent, or SG_CVT_E_FAILURE to indicate a general failure, free the buffer or re-use it as appropriate. If the call returns SG_CVT_E_AGAIN (you passed B_FALSE as the canwait argument) your data has not been sent, and you should retain it to try again later.
The following example shows a simple stream converter. It expects a stream of ASCII text characters, and outputs the stream with any uppercase characters replaced by their lowercase equivalents.
#include <SgCvt.h> #include <dmedia/dm_params.h> void CvtToLower ( void *arg ) { SgCvtConverterData *data = (SgCvtConverterData *) arg; SgCvtStatus s; char buf[BUFSIZ]; size_t nreq = BUFSIZ; unsigned int len=0; int start = 0; int i; /* Evaluation */ if (data->operation == SG_CVT_REQ_EVALUATE) { /* * In less trivial converters, we would check for * valid params in the context, but in this case all * we're doing is byte translation, so we can always * say yes. */ data->status_return = SG_CVT_E_ACCEPT; return; } /* * Conversion Loop. A similar construct will appear in * all streaming converters. The model is fetch data, * convert it and forward it, until we have forwarded the * end of stream, then jump out the loop. */ for (;;) { s = SgCvtGetData(data->context, nreq, buf, &len, NULL, B_TRUE); if (s == SG_CVT_E_FAILURE) { fprintf(stderr, “converter: failed to get data\n”); return; } if (s == SG_CVT_E_END_OF_STREAM) { SgCvtSendEndOfStream(data->context); break; } /*** start converter-specific part ***/ for (i=0; i<len; i++) buf[i] = tolower(buf[i]); /*** end converter-specific part ***/ s = SgCvtSendData(data->context, (void *)buf, len, NULL, B_TRUE); if (s == SG_CVT_E_FAILURE) { fprintf(stderr, “converter: failed to get data\n”); return; } start += (len); } /* * When we get here, this converter's work * is complete. Others in the same pipeline may * still be running, but that's irrelevant to us. * We simply return. If we were invoked in a dedicated * sproc “thread”, which is always the case for * streaming converters, this terminates it. */ return; } |
Note the above comment about other converters: Others in the same pipeline may still be running. It is important to remember that your converter is almost always invoked as a subprocess of the application. “Programming Constraints” lists some considerations when writing converter code.
GoldenGate converters reside in Dynamic Shared Object (DSO) libraries.
After you have written and tested your conversion function by calling it directly from a test program, you are ready package it as a GoldenGate converter.
This section covers the following topics:
Create a DSO for your converter. A simple Makefile (below) for the previous example, “Example of a Simple Stream Converter”, illustrates the compilation and linkage requirements for a GoldenGate DSO.
# # Makefile for GoldenGate Sample Converter DSO # CVTR = CvtToLower all: cc -c -I/usr/include/convert $(CVTR).c ld -no_unresolved -o libUserCvtrs.so -shared $(CVTR).o |
After you compile your converter, you must create a converter description file that identifies your converter to GoldenGate. You use this file to test your converter, and intimately to register it with GoldenGate. A simple example for the CvtToLower converter follows.
# # Lowercase Text Stream # Converter { Name: “CvtToLower” IOMethod: StreamToStream Input: “MIXEDCASE” InputLabel: “ASCII bytes, any case” Output: “LOWERCASE” OutputLabel: “ASCII bytes, lower case” Vendor: “SGI (Sample)” Version: “1.0” Description: “Lowercases chars in input stream” DSO: “/usr/people/fred/libFredsCvtrs.so” Function: “CvtToLower” } |
Make sure the DSO field is set to the full pathname for the DSO you have built.
The grammar of the converter description file is fairly simple. Three types of statements exist; they are identified by the keywords Parameter, Converter, and Pipeline. Table G-4 defines the statements.
Table G-4. Converter Description File Statements
Statement | Description |
---|---|
Parameter statement | Defines a single parameter |
Converter statement | Describes a converter and may include Parameter statements |
Pipeline statement | Defines a series of converters to be used together, and may contain both Converter and Parameter statements |
Some example descriptions follow. The easiest way to write a converter description file is to copy an existing one. You can use these examples, or copy entries from the default registry file, /var/GoldenGate/ConverterRegistry.
# # Lowercase Text Stream # Converter { Name: “CvtToLower” IOMethod: StreamToStream Input: “MIXEDCASE” InputLabel: “ASCII bytes, any case” Output: “LOWERCASE” OutputLabel: “ASCII bytes, lower case” Vendor: “SGI (Sample)” Version: “1.0” Description: “Lowercases chars in input stream” DSO: “/usr/people/fred/libFredsCvtrs.so” Function: “CvtToLower” } # # Windows BMP to XWD # Converter { Name: “BMP_FILE_TO_XWD_FILE” IOMethod: FileToFile Input: “BMP_FILE” InputLabel: “BMP_FILE” Output: “XWD_FILE” OutputLabel: “XWD_FILE” Vendor: “SGI” Version: “1.0” Description: “BMP_FILE to XWD_FILE” DSO: “libcvt_SGI.so” Function: “xwdout” } # # Windows BMP to Compuserv GIF-89, through JPEG (JFIF) # This isn't necessary, since the default converters # can go directly to GIF 89 from BMP, but it illustrates the # Pipeline syntax for chaining converters together. Pipeline { Name: “BMP_FILE_TO_GIF_89_FILE” IOMethod: FileToFile Input: “BMP_FILE” InputLabel: “Windows BMP” Output: “GIF_89_FILE” OutputLabel: “Compuserve GIF” Vendor: “SGI” Version: “1.0” Description: “Windows BMP to GIF, via JPG” Converter { Name: “BMP_FILE_TO_JFIF_FILE” IOMethod: FileToFile Input: “BMP_FILE” InputLabel: “BMP_FILE” Output: “JFIF_FILE” OutputLabel: “JFIF_FILE” Vendor: “SGI” Version: “1.0” Description: “BMP_FILE to JFIF_FILE” DSO: “libcvt_SGI.so” Function: “jfifout” } Converter { Name: “JFIF_FILE_TO_GIF_89_FILE” IOMethod: FileToFile Input: “JFIF_FILE” InputLabel: “JFIF_FILE” Output: “GIF_89_FILE” OutputLabel: “GIF_89_FILE” Vendor: “SGI” Version: “1.0” Description: “BMP_FILE to GIF_89_FILE” DSO: “libcvt_SGI.so” Function: “gifout” } } |
To test your converter, first verify that your converter description file is valid and does not cause the GoldenGate built-in registry parser to fail.
Set the environment variable CVT_REGISTRY_OVERRIDE to the full pathname of the converter description file you just created:
setenv CVT_REGISTRY_OVERRIDE /usr/people/fred/my_registry.cdf |
Then run a test program that will exercise the parser. The gg_query demo program that comes with GoldenGate is good for this. Find it in /usr/share/src/GoldenGate (if you haven't already done so, install the demo programs from your IRIX distribution media). Copy the demo programs to your own directory, go into the Query subdirectory, and type make. Then execute the gg_query program. The output should look like this:
Converter (CvtToLower): method: Stream To Stream input: MIXEDCASE (ASCII bytes, any case) output: LOWERCASE (ASCII bytes, lower case) vendor: SGI (Sample) version: 1.0 descr.: Lowercases chars in input stream DSOname: /usr/people/fred/libFredsCvtrs.so Function: CvtToLower |
If you see an error message, go back and check that your converter description file is valid, checking especially that all string values are properly quoted. Also check that the GoldenGate software is properly installed by unsetting the CVT_REGISTRY_OVERRIDE variable and re-executing the gg_query program. It should list the default converters installed on the system (over 100 of these exist).
Once the test runs successfully, you are ready to try executing your converter. You can use your own program, or the demo programs in the ConvertFile and ConvertStream directories to do this. Each program prints a help message describing its arguments if you run it with no arguments.
After you are satisfied that your converter works when executed via GoldenGate, you are ready to make it available to other applications on the system. Unset the CVT_REGISTRY_OVERRIDE variable; you are finished unit-testing your converter.
To register your converter, you must add your converter description file to the system registry.
The system registry is a text file that uses the same syntax as your converter description file. Just edit the file /var/GoldenGate/ConverterRegistry (you must be a privileged user to do this) and add your entry wherever you like.
Look at the attributes of the converters already registered. If there are potential clashes with your converter, you may wish to insert your converter closer to the beginning of the registry. Some applications may decide to convert using the first converter they find that appears to satisfy their requirements, rather than evaluating the alternatives. If you want to make sure this kind of application executes your converter rather than another one that could do that same conversion, insert your entry closer to the beginning of the file.
The most important fields are those that the service uses to locate the executable converter module: the DSO name and the Function name. The other fields are primarily for display by administration tools, and for applications to query the registry. The Input and Output fields are strings that must exactly match the format names that applications will use to search for converters. For instance, where there are naming conventions such as ICCCM target names, these should be used exactly.
See “Supported Target Formats” in Chapter 7 for a list of standard input and output formats supported by the default converters supplied with GoldenGate.
Parameters can be one of two types: Constraint or Programmable.
Constraint parameters are used to specify constant values for a data attribute in the description file. When you see a constraint parameter, it means that this converter always sets the corresponding data attribute to the stored value, overriding its current value in the input.
Programmable parameters are used to specify parameters that are set at runtime based upon the requested input and output parameters. Programmable parameters are used to pass a runtime parameter to one of the stages of a pipeline. For example, if you have a two-stage pipeline designed to scale an SGI image to an arbitrary size, then convert it to JPEG, you want to pass one of the output parameters (the required output size) to the first stage of the converter. You do this by specifying a programmable parameter for the first stage.
Make sure your installation copies the DSO containing your converter to the standard location for converter DSOs: /usr/lib/convert. If you install your library there, you can use a relative DSO name in your converter description file. If you install anywhere else, you must use a full path name in the registry to ensure that the service will find your converter.
See the GoldenGate Release Notes (type relnotes goldengate) for information about installation.
This section presents annotated sample code for two different converters:
The first example, “A Simple StreamToStream Converter - UpperCase”, directly modifies data flowing through it.
The second example, “A FileToFile Converter - UNIX Man Page File to HTML File”, illustrates two techniques. First it serves as a basic template for FileToFile converters, and second it shows how you can wrap an external IRIX command to make it available as a GoldenGate converter.
This type of converter can often offer the best performance in many circumstances, because all the knowledge of the conversion operation is in the converter itself, and because it typically does not need to access the filesystem to achieve conversion. It is appropriate when the data format is naturally streamable, such as ASCII text or other self-identifying or raw data.
The converter used in this example performs a simple mapping of mixed-case text to uppercase text. The converter-specific parts are clearly marked. These are the lines that you will replace with your own task-specific conversion code. The remainder is boilerplate code that can be re-used in many different converters.
/* converter function */ void UpperCase ( void *arg ) { SgCvtConverterData *data = (SgCvtConverterData *) arg; SgCvtStatus s; |
The next 2 lines are somewhat task-specific. Your converter should use a buffer size appropriate to the data type and the task. Careful selection of a buffer size will yield better performance in many cases. For instance, if your converters needs to operate on audio or movie “frames,” then you may choose to read and write buffers that represent whole numbers of frames.
char buf[BUFSIZ]; size_t nreq = BUFSIZ; unsigned int len=0; int start = 0; int i; |
The next part is the Evaluation section. Our converter operates on a byte stream: if a byte represents a lower-case character in the current locale, we are going to uppercase it. Otherwise it passes through untouched. It is appropriate for this converter to accept any stream; it does not need to evaluate parameters.
/* Evaluation */ if (data->operation == SG_CVT_REQ_EVALUATE) { data->status_return = SG_CVT_E_ACCEPT; return; } |
The following loop does the conversion, one block at a time. The loop terminates when the end of stream is detected.
/* Conversion */ for (;;) { s = SgCvtGetData(data->context, nreq, buf, &len, NULL, B_TRUE); if (s == SG_CVT_E_END_OF_STREAM) { SgCvtSendEndOfStream(data->context); break; } |
These two lines show the entire task-specific code requirements for the uppercase text converter. Your converter will substitute its own conversion-specific code for these lines. The model is the same in each case; the converter generates a buffer to be sent from the buffer it has received, by applying a specific conversion algorithm.
for (i=0; i<len; i++) buf[i] = toupper(buf[i]); |
When the new buffer has been generated, your converter sends it into the pipeline. In this example, the data was converted in place. Sometimes that is not possible, because the converted data will not fit in the original buffer. In these cases, your converter may allocate, populate, send, then free a dynamic buffer each time through the loop.
s = SgCvtSendData(data->context, (void *)buf, len, NULL, B_TRUE); start += (len); } |
After sending all the converted data, and calling SgCvtSendEndOfStream, your converter can return. You should always set the status_return field.
data->status_return = SG_CVT_E_SUCCESS; return; } |
Often, to convert data from one application into a form usable by another, you need to save the data to a file in one format, convert it using an IRIX command-line translator program, then open the new file using the application you want to use.
Applications using components that are integrated with GoldenGate can eliminate the intermediate end-user steps. The same external translator command can be packaged as a GoldenGate converter and invoked automatically on behalf of the user.
It is quite straightforward to integrate an existing IRIX command with GoldenGate. Your main task is to write a function that the service can invoke, which constructs from its parameters a command line for the translator program.
The converter described here provides GoldenGate access to a command named man2html, which converts troff source files for UNIX man pages into HTML files that can be viewed using a Web browser. It can be used by a CGI script that implements an online help system for remote users running Web browsers.
The command itself takes one argument: the input file name. It writes its output to stdout. The job of our converter is to make this interface look like any other GoldenGate converter.
Converter functions always require the SgCvt.h header file and always have the have the same calling convention:
#include <libgen.h> #include <SgCvt.h> void ManToHtml(void *arg) { |
First, cast the data argument to the right type.
SgCvtConverterData *data = (SgCvtConverterData *) arg; |
Then, define some other local variables. Most of these are the same in every converter of this type that you write.
SgCvtStatus s; SgCvtContextInfo ctx_info; char cmd[BUFSIZ]; int sys_status = 0; char *cmdpath; |
Next, handle converter evaluation requests. This converter is very simplistic: there are no parameters, and it just ACCEPTs the request. In real converters, always provide proper evaluation of any parameters, especially if it is expensive for the application to try the conversion and fail.
/* Evaluation */ if (data->operation == SG_CVT_REQ_EVALUATE) { data->status_return = SG_CVT_E_ACCEPT; return; } |
The remaining code handles conversion requests. Note that it is never reached unless the caller requests conversion because the evaluate section has its own return statement.
The first thing to do is check that the program you are wrapping is installed and that you can execute it.
/* conversion */ cmdpath = pathfind(getenv (“PATH”), “man2html”, “rx”); if (cmdpath == NULL) { data->status_return = SG_CVT_E_MISSING_COMMAND; return; } |
Then, extract the input and output filenames from the conversion context.
(void) SgCvtGetContextInfo(data->context, SG_CVT_INFO_INPUT_FILE | SG_CVT_INFO_OUTPUT_FILE, &ctx_info); |
At this point you know everything needed to construct the command you are going to execute. You use the UNIX system(2) call to execute the conversion, so the next step is to create the command line.
sprintf(cmd, “%s %s > %s 2> /dev/null”, cmdpath, ctx_info.input_file, ctx_info.output_file); |
Finally, execute the command, and set the return status to indicate whether it worked before returning.
sys_status = system(cmd); data->status_return = sys_status ? SG_CVT_E_FAILURE : SG_CVT_E_SUCCESS; return; } |
Note that your converter is normally run as a subprocess of the invoking application. You should not call exit(2) to terminate your converter; you should simply return, allowing GoldenGate and the operating system to take care of managing conversion threads.