March 28, 2012 VO/IRAF Integration Overview Contents: Introduction IRAF System Enhancements URL Support VOTable Support System File Cache Enhanced @-file templates VO-CL - A New CL with VO Features VOClient - Client-Side VO Interfaces SAMP - Desktop Messaging Receiving Messages in the CL Sending Messages from the CL Command Mode Summary VO External Package Toolkit Tasks Jargon Words References Tutorial Finding VO Data Resources Accessing Catalog Data Accessing Image Data Virtual Observatory FAQ ============================================================================== ============ Introduction ============ The VO-IRAF Integration project is intended to incorporate Virtual Observatory (VO) technologies and services into the IRAF analysis environment in a seamless and natural (from the IRAF perspective) manner. Because of the size of the IRAF system (both the core system itself and the dozens of layered packages), a systematic approach to this integration must be at the core of the development in order to VO-enable as many applications as possible. At the same time, IRAF tasks that make use of VO-specific capabilities must be developed to provide a basis for new functionality and science capabilities at the user level (in some cases, these will be developed by users themselves). At the IRAF system level, we have merged concepts common to both IRAF and VO in a form that is natural for the IRAF environment. For example, a VOTable (the XML document standard common to all VO services) of image or spectral data-service (SIA, Simple Image Access and SSA, Simple Spectral Access) query results containing access references (URLs) to data can be thought of as a list of remote images, this is not unlike the '@file' list of images now allowed by many tasks to process a list of local images; Similarly, use of URLs for the transparent access to remote data is not unlike what is now done with 'iraf networking'. By allowing a VOTable to have equal status with current list processing capabilities in the system, and by providing access to remote data as if they were already local, we can hide details (and complexities) of the VO infrastructure in a way which still makes these services and data useful. Likewise, we would like to expose VO resources to the user in a way that provides a simple way to access and use these data to augment an astronomer's understanding of their data. For every image acquired at a 'real' observatory, we can guarantee that somewhere in the VO there are catalogs or spectra of objects in the same field (or of the primary target), images of that field in other bandpasses, or services that can describe to moving objects at the time an image was acquired or which link to published/aggregated data in the field. Image metadata typically contains all the information necessary to form a VO query for other data, e.g. the WCS provides the image center/resolution/footprint, filter information to specify spectral coverage, time keywords to specify temporal coverage, and so on. Where a typical VO data query is constructed by specifying these parameters individually, users would prefer to simply query for data that "matches" the image they have and leave it to the application to derive and format the parameters for a query. This again hides the details of VO workings from the user but seamlessly allows the user to access catalogs and VO services (e.g. the list of bright X-Ray sources in the field, 21-cm radio imaging, etc) that supplement the data obtained at the telescope. The initial IRAF v2.16 release of VO features provides the system infrastructure needed (e.g. URL and VOTable support) and a basic toolbox of utilities needed for building VO science applications. The VO external package will be expanded with additional utilities and science applications in future releases. Because system changes such as URL support affect all task binaries (and external packages linked with v2.16), it is likely that bug fixes will similarly affect all binaries. For this reason (and as well to release new tasks as they become available), a simplified patch system is also part of the v2.16 release. ======================== IRAF System Enhancements ======================== VO data and services are, almost by definition, remote. Interfaces and protocols are defined by the IVOA (International Virtual Observatory Alliance, and international collaboration of VO projects) for data query and access as well as for standard VO services such as the Registry (a form of master index of VO Resources); custom interfaces (e.g. to the NED data service or a compute service doing catalog cross-comparison) also exist within the VO but are not standardized in the same way. Throughout the VO the use of URLs and VOTables are ubiquitous for querying, exchanging or referring to remote data resources; for the level of integration we desire, remote data must be as easily accessible as local resources. Further, it shouldn't be necessary for an astronomer to know the details of these VO protocols in order to use them. In order to support URLs and VOTables in a systematic way, changes to system interfaces (e.g. image and file I/O) were required so that all tasks would benefit. Details of these changes are described below, but it should be noted that these changes enhance the system in non-VO contexts as well. URL Support ----------- Data services are typically queried using a simple HTTP-GET URL containing parameters for the position on the sky (equatorial ICRS) and the size of the search (also in decimal degrees); in some cases a service may use another form of web-service invocation (e.g. the Sesame name resolver) but this is handled internally and not generally exposed to the IRAF user. VO catalog services return the results directly in the form of a VOTable, image and spectral services return a table of data found with an 'access reference' column, i.e. a URL to the FITS file (this is to allow the client to select images to access rather than download many unnecessary files). Therefore, for both querying-for and retrieving VO data, support for URLs at the application level is essential. In VO-IRAF, a URL shares equal status with an explicit file path, an iraf logical path or an IRAF networking path (for input data only). That is to say, the commands vocl> imstat http://iraf.noao.edu/votest/dpix.fits or vocl> s1 = "http://iraf.noao.edu/votest/dpix.fits" vocl> imstat (s1) vocl> display (s1, frame=1) or vocl> imcopy http://iraf.noao.edu/votest/dpix.fits foo.fits are all equal ways to run the IMSTAT task on the image at the target URL. This support extends to query URLs as well, i.e. if the URL return a FITS image or other file appropriate for the task, the task will execute as if this were a local file reference. Internally, URLs are retrieved and placed into the system file cache (see below) and the task is presented with the local cache file reference. The download introduces a (perhaps slight, sometimes lengthy) delay before the task begins, however the operation on the file is transparent to the task. Subsequent uses of the same URL map immediately to the cached file and thus avoid a repeated download delay. Use of a query URL is one way to combine the access to a remote file with its use in the task and avoid a separate download stage, i.e. one can (for all practical purposes) operate directly on images at remote locations. This capability is not restricted to VO services, data from any archive that has an access URL (e.g. public data from non-VO archives or a collaborator's personal URL to an interesting image) may be used. Caveats: - URLs are supported ONLY for input data and may not be used for output parameters to POST results of a task to a remote location. - Data accessed by URL may be modified by a task, but users should be aware it is only the locally cached file that is modified. Continued use of the URL in a script will refer to the local copy and not modify the target URL. - Cached URL data IS NOT checked to see if the target URL is newer than the cached data, it is assumed that these files are relatively static compared to what has been previously downloaded. - URLs as input parameters are assumed to point to a remote resource of the type expected by the task, i.e. if the task expects an IRAF image file it will throw the same error as if presented with a local text/XMl file instead. VOTable Support ---------------- A 'VOTable' is an XML document standard used throughout the VO for exchanging data, it provides both metadata and data in a single file in much the same way a FITS file might. As the name suggests, VOTable is a tabular data format, in a VO context it provides either tabular data directly (e.g. from a Cone Search or TAP query), or a result table for a query that may contain an access reference (i.e a URL) to the target (image or spectra) data requiring a separate step to actually access. The separation of query-result from direct-download for image/spectral data is meant to allow client applications the opportunity to parse which of the results are most appropriate/useful before beginning what could be a lengthy download process. A VO client application can then perform a general query on position, but download only those data that refer to object images (and not calibration/raw data) or a spectrum with a desired resolution limit, or other constraints present in the query-result table. In IRAF v2.16, the XML VOTable document is automatically converted to a format (specifically, a FITS BINTABLE) that IRAF tasks can parse, this is true whether as a result of a URL-query or a local reference to a file. Internally, the VOTable is converted and put into the system file cache, that converted file is then given to the task. The converted table is then used as the task expects, i.e. a command like vocl> tinfo votable.xml treats the VOTable input file the same as it would the a local FITS BINTABLE, the only difference being that the conversion is performed atuomatically. A number of tasks in the VOTOOLS package are based directly on the VOTable parser interface and will operate on the VOTable without conversion. Because there is no conversion, these tasks will generally only use the VOTable as input without modification in order to avoid conflicts with the cached version of the converted table used by IRAF tasks. For tasks that might modify the file, use of a different output filename will be enforced by the task. VOTOOLS Applications: The following applications are available specifically to deal with VOTable documents directly: VOTABLE UTILITY TOOLS votcopy - Copy a VOTable to another format votget - Download data referenced in a VOTable votpos - Extract the main positional columns from a VOTable votsize - Get the size of a VOTable TABLE UTILITIES colbyid - Identify VOTable column by ID attribute colbyucd - Identify VOTable column by UCD attribute colbyname - Identify VOTable column by NAME attribute System File Cache ----------------- Enhanced @-file templates ------------------------- VO-CL - A New CL with VO Features ================================= VOClient - Client-Side VO Interfaces ==================================== SAMP - Desktop Messaging ======================== The SAMP interface in IRAF is implemented in two modes: A "Command Mode" provides a natural command-line interface to either control the SAMP messaging (e.g. to enable/disable messaging), or to send messages manually. As an example, cl> samp start # start SAMP messaging on <-- shows status This also takes advantage CL-specific features such as logical directories or support for sexagesimal value. For instance: cl> samp loadImage data$foo.fits # broadcast an image load message ok cl> samp pointAt (15 * 10:23:01) 34:12:45 to=aladin ok In the first case the 'data$' logical dir is converted internally so the SAMP application is given the expected URI, in the second case the arguments are expected to be in decimal degrees and converting RA is done trivially. The second form, known as "Program Mode", provides the same functionality in the form of CL builtin functions more suited for CL scripting. For example, if (sampStatus("on") != "on") { error (0, "Cannot enable SAMP messaging") } else { if (sampLoadImage (img, "aladin") != "ok") { error (0, "Cannot load image " // img) } } A complete list of both the Command and Program mode functions is given in the Appendix below. Explicitly enabling the SAMP message (e.g. with "samp start") is not necessary to send messages to other applications, any 'samp' command that requires a SAMP Hub will attempt to start a connection automatically. However, for the CL to receive messages from other applications the SAMP interface must be enabled. This can be done explicitly as above, by setting the "samp_onstart" CL environment variable in the hlib$zzsetenv.def file, or by uncommenting the "samp on" command in the login.cl file. Receiving Messages in the CL ---------------------------- By default, the CL will subscribe only to the following SAMP mtypes: client.cmd.exec # execute a command string client.env.set # set and environment variable client.env.get # get and environment variable (*) client.param.set # set a task parameter value client.param.get # get a task parameter value Except for the client.env.get mtype, these are all newly defined mtypes that may not be available to other SAMP applications. Testing these messages can be done with the samp library tasks or from another CL session. Additional message types can be received by defining handlers for each mtype, for example cl> samp handler image.load.fits "imstat $url" or cl> =sampHandler ("image.load.fits", "imstat $url") This will subscribe the current session to the 'image.load.fits' mtype, and when received will execute the IMSTAT command on the named URL. The '$url' in the command string is replaced by the value of that parameter sent in the message. The mtype parameter names are defined in the Appendix below. Proprietary messages can be defined using the same mechanism. For instance cl> samp handler pipeline.event "imcopy $url image.fits" will create a handler for the 'pipeline.event' mtype and execute the associated command. Another client could send a message of this type regardless of whether this is an mtype approved by the IVOA. Calling the sampHandler (or 'samp handler') function with no arguments will print the list of user-defined handlers, supplying only the 'mtype' argument will print the handler defined for just that mtype (or a null string). By default a SAMP-enabled CL session will be known as 'IRAF' to other applications, this can lead to a name conflict when sending directed messages. To solve this, a session can define it's own name to other applications using any of the following: cl> samp name iraf2 cl> samp meta samp.name iraf2 cl> =sampName ("iraf2") cl> =sampMetadata ("samp.name", "iraf2") The sampMetadata() function (or 'samp meta' command) can declare arbitrary metadata about the session that can be discovered by other apps, the name in this case is provided as a convenience function and will cause the name to be updated immediately. Sending Messages from the CL ---------------------------- Sending messages from the CL can be done using either the command or program form of a command, high-level functions are implemented for the most common messages (e.g. to 'load an image') as well as one low-level method for sending arbitrary messages. For example, cl> samp loadImage http://foo.edu/test.fits # load an image cl> samp loadVOTable images.xml # load a VOTable cl> samp loadFITS imtab.fits # load a FITS bintable Or the low-level 'send' command as in cl> samp send pipeline.event url=http://foo.edu/test.fits or cl> =sampSend ("pipeline.event", "url=http://foo.edu/test.fits") In this case the first argument is always the mtype, remaining arguments are of the form "=" and may be used to define arbitrary parameters of the message. To send a message to a specific application, the argument "to=" may be appended to these commands, e.g. cl> samp loadFITS test.fits to=topcat # load table in Topcat When using the program mode the recipent application name is one of the optional parameters, e.g. cl> =sampLoadImage ("image.fits") # broadcast to all apps cl> =sampLoadImage ("image.fits", "aladin") # send to aladin Command Mode Summary -------------------- # The following command is a builtin but doesn't push anything on the # result stack, instead result values are printed to the stdout/stderr. # The intent is to provide a simple command interface without requiring # the function syntax. samp status cl_Samp samp on|start : samp off|stop : samp restart : samp name [] : samp trace [] : samp access [] : samp handler [ ] : samp meta [ ] : samp send [ ....] : samp exec : samp pointAt : samp setenv : samp getenv : samp setparam : samp getparam : samp loadImage : samp loadVOTable : samp loadFITS : samp showRow [] [] : samp selectRows [] [] ,,.... : Program Mode Summary -------------------- # The following functions do push a result on the stack. This allows # scripts to be written to check the return value before continuing. on|off = sampStatus ( [on|off|restart] ) func_sampStatus yes|no = sampAccess (appName) func_sampAccess ok|name = sampName ([name]) func_sampName ok|val = sampMetadata ([name, [val]]) func_sampMetadata yes|no = sampHandler (mtype, cmd) func_sampAddHandler ok|err = sampLoadImage (url[, to, id, name]) func_sampLoadImage ok|err = sampLoadVOTable (url[, to, id, name]) func_sampLoadVOTable ok|err = sampLoadFITS (url[, to, id, name]) func_sampLoadFITS ok|err = sampLoadSpec (url[, to, id, name]) func_sampLoadImage ok|err = sampLoadBibcode (bibcode[, to]) func_sampLoadBibcode ok|err = sampLoadResource (url[, to, meta, id, name]) func_sampLoadResource ok|err = sampShowRow (url, id, row[, to]) func_sampShowRow ok|err = sampSelectRowList (url, id, *row[, to]) func_sampSelectRowList ok|err = sampPointAt (ra, dec[, to]) func_sampPointAt ok|err = sampCmdExec (cmd[, to]) func_sampCmdExec ok|err = sampEnvSet (par, val[, to]) func_sampEnvSet val|err = sampEnvGet (par[, to]) func_sampEnvGet ok|err = sampParamSet (par, val[, to]) func_sampParamSet val|err = sampParamGet (par[, to]) func_sampParamGet -------------------------------------------------------------------------------- Command Function on|start Y . off|stop Y . restart Y . status Y . access Y . handler Y . metadata Y . send Y . loadImage Y . loadVOTable Y . loadFITS Y . cmdExec Y . envGet Y . envSet Y . paramGet Y . paramSet Y . pointAt Y . showRow Y . selectRows Y . resourceLoad . . specLoad . . bibcodeLoad . . ================================================================================ VO External Package =================== Toolkit Tasks ------------- Jargon Words ============= DAL Data Access Layer -- A general term referring to all VO data services. IVOA International Virtual Observatory Alliance -- An international collab- oration of national VO projects. SAMP Simple Application Messaging Protocol -- An IVOA standard for desktop messaging. SCS Simple Cone Search -- An IVOA protocol for simple catalog queries. SIA Simple Image Access --An IVOA protocol for image data queries. SSA Simple Spectral Access --An IVOA protocol for spectral data queries. VAO Virtual Astronomical Observatory -- The US VO project VO Virtual Observatory VOClient An interface library used to handle the client-side interface to VO services. VOEvent A VO working group to define standards for transient event alerts. VOTable An XML document standard for VO tablular data. This is also used for status returns from DAL services. References ========== IVOA Document Repository: http://www.ivoa.net/Documents Simple Application Messaging Protocol v1.3: http://www.ivoa.net/Documents/SAMP/ Simple Cone Search v1.03: http://www.ivoa.net/Documents/latest/ConeSearch.html Simple Image Access Specification v1.0: http://www.ivoa.net/Documents/SIA/ VOTable Format Definition v1.2: http://www.ivoa.net/Documents/VOTable/ Tutorial ======== Finding VO Data Resources ------------------------- Accessing Catalog Data ---------------------- Accessing Image Data -------------------- Virtual Observatory FAQ =======================