Data Access (pyvo.dal)

This subpackage provides access to the various data servies in the VO.

Getting started

Service objects are created with the service url and provide service-specific metadata.

>>> service = vo.dal.SIAService("http://dc.zah.uni-heidelberg.de/lswscans/res/positions/siap/siap.xml")
>>> print(service.description)

They provide a search method with varying standard parameters for submitting queries.

>>> resultset = service.search(pos=pos, size=size)

which returns a resultset.

Individual services may define additional, custom parameters. You can pass these to the search method as (case-insensitive) keyword arguments.

Call the method describe to print human-readable service metadata. You most likely want to use this in a notebook session or similar before actually querying the service.

See Services for a explanation of the different interfaces.

Astrometrical parameters

Most services expose the astrometrical parameters pos and size for which PyVO accept SkyCoord or Quantity objects as well as any other sequence containing right ascension and declination in degrees, which are converted to the standard coordinate frame (in the VO, that usually is ICRS) in the standard units (always degrees in the VO) before they are submitted to the service.

Also, SkyCoord can be used to lookup names of astronomical objects you are searching for.

>>> import pyvo as vo
>>> from astropy.coordinates import SkyCoord
>>> from astropy.units import Quantity
>>>
>>> pos = SkyCoord.from_name('NGC 4993')
>>> size = Quantity(0.5, unit="deg")

See Astronomical Coordinate Systems (astropy.coordinates) and Units and Quantities (astropy.units) for details.

The Quantity object is also suitable for any other astrometrical parameter, such as waveband ranges.

Some services also accept Time as time parameter.

>>> from astropy.time import Time
>>> time = Time(
>>>     ('2015-01-01T00:00:00', '2018-01-01T00:00:00'),
>>>     format='isot', scale='utc'
>>> )

See Time and Dates (astropy.time) for explanation.

Verbosity

Several VO protocols have the notion of “verbosity”, where 1 means “minimal set of columns”, 2 means “columns most users can work with” and 3 ”everything including exotic items”. Query functions accept these values in the verbosity parameter. The exact semantics are service-specific.

Availability and capabilities

VO services should offer some standard ”support” interfaces specified in VOSI. In pyVO, the information obtained from these endpoints can be obtained from some service attributes.

For availability (i.e., is the service up and running?), this is available and up_since

Capabilities describe specific pieces of functionality (such as “this is a spectral search”) and further metadata (such as ”this service will never return more than 10000 rows”).

This information is contained in the datastructure CapabilitiesFile available through capabilities.

Exceptions

See pyvo.dal.exceptions.

Services

There are five types of services with different purposes but a similiar interface available.

Table Access Protocol

Unlike the other services, this one works with tables queryable by an sql-ish language called ADQL instead of predefined search constraints.

>>> tap_service = vo.dal.TAPService("http://dc.g-vo.org/tap")
>>> tap_results = tap_service.search("SELECT TOP 10 * FROM ivoa.obscore")

As a sanity precaution, most services have some default limit of how many rows they will return before overflowing:

>>> print(tap_service.maxrec)

To retrieve more rows than that (often conservative) default limit, you must override maxrec in the call to search:

>>> tap_results = tap_service.search(
>>>     "SELECT * FROM ivoa.obscore", maxrec=100000)

Services will not let you raise maxrec beyond the hard match limit:

>>> print(tap_service.hardlimit)

A list of the tables and the columns within them is available in the TAPService’s tables attribute by using it as an iterator or calling it’s describe() method for a human-readable summary.

Uploads

Some TAP services allow you to upload your own tables to make them accessible in queries.

For this the various query methods have a uploads keyword, which accepts a dictionary of table name and content.

The mechanism behind this parameter is smart enough to distinct between various types of content, either a str pointing to a local file or a file-like object, a Table or DALResults for an inline upload, or a url str pointing to a remote resource.

The uploaded tables will be available as TAP_UPLOAD.name.

Note

The supported upload methods are available under upload_methods().

Simple Image Access

Like the name says, this service serves astronomical images.

Basic queries are done with the pos and size parameters described in Astrometrical parameters, with size being the rectangular region around pos.

>>> pos = SkyCoord.from_name('Eta Carina')
>>> size = Quantity(0.5, unit="deg")
>>> sia_service = vo.dal.SIAService("http://dc.zah.uni-heidelberg.de/hppunion/q/im/siap.xml")
>>> sia_results = sia_service.search(pos=pos, size=size)

The dataset format, ‘all’ by default, can be specified:

>>> sia_results = sia_service.search(pos=pos, size=size, format='graphics')

This would return all graphical image formats (png, jpeg, gif) available. Other possible values are image/* mimetypes, or metadata, which returns no image at all but instead a declaration of the additional parameters supported by the given service.

The intersect argument (defaulting to OVERLAPS) lets a program specify the desired relationship between the region of interest and the coverage of the images (case-insensitively):

>>> sia_results = sia_service.search(pos=pos, size=size, intersect='covers')
Available values:
COVERS select images that completely cover the search region
ENCLOSED select images that are complete enclosed by the region
OVERLAPS select any image that overlaps with the search region
CENTER select images whose center is within the search region

This service exposes the verbosity parameter

Simple Spectrum Access

Access to (one-dimensional) spectra resembles image access, with some subtile differences:

The size parameter is called diameter here, and hence the search region is always circular with pos as center:

>>> ssa_service = vo.dal.SSAService("http://www.isdc.unige.ch/vo-services/lc")
>>> ssa_results = ssa_service.search(pos=pos, diameter=size)

SSA queries can be further constrained by the band and time parameters.

>>> ssa_results = ssa_service.search(
>>>     pos=pos, diameter=size,
>>>     time=time, band=Quantity((1e-13, 1e-12), unit="meter")
>>> )

Simple Line Access

This service let you query for spectral lines in a certain wavelength range. The unit of the values is meters, but any unit may be specified using Quantity.

Jobs

Some services, most notably TAP ones, allow asynchronous operation (i.e., you submit a job, receive a URL where to check for updates, and then can go away) using a VO standard called UWS.

These have a submit_job method, which has the same parameters as their search but start a server-side job instead of waiting for the result to return.

This is particulary useful for longer-running queries or when you want to run several queries in parallel from one script.

Note

It is good practice to test the query with a maxrec constraint first.

When you invoke submit job you will get a job object.

>>> async_srv = vo.dal.TAPService("http://dc.g-vo.org/tap")
>>> job = async_srv.submit_job("SELECT * FROM ivoa.obscore")

Note

Currently, only pyvo.dal.TAPService supports server-side jobs.

This job is not yet running yet. To start it invoke run

>>> job.run()

Get the current job phase:

>>> print(job.phase)
RUN

Maximum run time in seconds is available and can be changed with execution_duration

>>> print(job.execution_duration)
3600
>>> job.execution_duration = 7200

Obtaining the job url, which is needed to reconstruct the job at a later point:

>>> job_url = job.url
>>> job = vo.dal.tap.AsyncTAPJob(job_url)

Besides run there are also several other job control methods:

Note

Usually the service deletes the job after a certain time, but it is a good practice to delete it manually when done.

The destruction time can be obtained and changed with destruction

Also, pyvo.dal.AsyncTAPJob works as a context manager which takes care of this automatically:

>>> with async_srv.submit_job("SELECT * FROM ivoa.obscore") as job:
>>>     job.run()
>>> print('Job deleted!')

Check for errors in the job execution:

>>> job.raise_if_error()

If the execution was successful, the resultset can be obtained using fetch_result()

The result url is available under result_uri

Resultsets and Records

Resultsets contain primarily tabular data and might also provide binary datasets and/or access to additional data services.

To obtain the names of the columns in a service response, write:

>>> print(resultset.fieldnames)

Rich metadata equivalent to what is found in VOTables (including unit, ucd, utype, and xtype) is available through resultset’s getdesc() method:

>>>  print(resultset.getdesc("accref").ucd)

Note

Two convenience functions let you retrieve columns of a specific physics (by UCD) or with a particular legacy data model annotation (by utype), like this:

>>> fieldname = resultset.fieldname_with_ucd('phot.mag;em.opt.V')
>>> fieldname = resultset.fieldname_with_utype('Access.Reference')

Iterating over a resultset gives the rows in the result:

>>> for row in resultset:
>>>     print row['accref']
...

The total number of rows in the answer is available as its len():

>>> print(len(resultset))
9

As with general numpy arrays, accessing individual columns via names gives an array of all of their values:

>>> column = resultset['accref']

whereas integers retrieve columns:

>>> row = resultset[0]

and both combined gives a single value:

>>> value = resultset['accref', 0]

Row objects may expose certain key columns as properties. See the corresponding API spec listed below for details.

Multiple datasets

PyVO supports multiple datasets exposed on record level through the datalink. To get an iterator yielding specific datasets, call pyvo.dal.adhoc.DatalinkResults.bysemantics() with the identifier identifying the dataset you want it to return.

>>> preview = next(row.getdatalink().bysemantics('#preview')).getdataset()

Note

Since the creation of datalink objects requires a network roundtrip, it is recommended to call getdatalink only once.

Of course one can also build a datalink object from it’s url.

>>> datalink = DatalinkResults.from_result_url(url)

Server-side processing

Some services support the server-side processing of record datasets. This includes spatial cutouts for 2d-images, reducing of spectra to a certain waveband range, and many more depending on the service.

SODA

SODA is a service with predefined parameters, available on row-level through pyvo.dal.adhoc.SodaRecordMixin.processed() which exposes a set of parameters who are dependend on the type of service.

  • circle – a sequence (degrees) or astropy.units.Quantity of longitude, latitude and radius
  • range – a sequence (degrees) or astropy.units.Quantity of two longitude values and two latitude values describing a rectangle.
  • polygon – multiple pairs of longitude and latitude points
  • band – a sequence of two values (meters) or astropy.units.Quantity with two bandwitdh values. The right sort order will be ensured if converting from frequency to wavelength.

Interoperabillity over SAMP

Tables and datasets can be send to other astronomical applications, providing they have support for SAMP (Simple Application Messaging Protocol).

You can either broadcast whole tables by calling broadcast_samp on the resultset or a single product (image, spectrum) by calling this method on the SIA or SSA record.

Note

Don’t forget to start the application and make sure there is a runnung SAMP Hub.

Underlying data structures

PyVO also allows access to underlying data structures.

The astropy data class astropy.table.Table is accessible with the method pyvo.dal.DALResults.to_table(), following astropy naming conventions.

If you want to work with the XML data structures astropy.io.votable.tree.VOTableFile or astropy.io.votable.tree.Table, they are accessible by the attributes pyvo.dal.DALResults.resultstable and pyvo.dal.DALResults.votable, respectively.

Reference/API

pyvo.dal Package

Functions

imagesearch(url, pos[, size, format, …]) submit a simple SIA query that requests images overlapping a given region
spectrumsearch(baseurl[, pos, diameter, …]) submit a simple SSA query that requests spectra overlapping a given region
linesearch(baseurl, wavelength, **keywords) submit a simple SLA query that requests spectral lines within a wavelength range
conesearch(url, pos[, radius, verbosity]) submit a simple Cone Search query that requests objects or observations whose positions fall within some distance from a search position.
tablesearch(url, query[, language, maxrec, …]) submit a Table Access query that returns rows matching the criteria given.

Classes

DALService(baseurl) an abstract base class representing a DAL service located a particular endpoint.
SIAService(baseurl) a representation of an SIA service
SSAService(baseurl) a representation of an SSA service
SLAService(baseurl) a representation of an spectral line catalog (SLA) service
SCSService(baseurl) a representation of a Cone Search service
TAPService(baseurl) a representation of a Table Access Protocol service
DALQuery(baseurl, **keywords) a class for preparing a query to a particular service.
SIAQuery(baseurl[, pos, size, format, …]) a class for preparing an query to an SIA service.
SSAQuery(baseurl[, pos, diameter, band, …]) a class for preparing an query to an SSA service.
SLAQuery(baseurl[, wavelength, request]) a class for preparing an query to an SLA service.
SCSQuery(baseurl[, pos, radius, verbosity]) a class for preparing an query to a Cone Search service.
TAPQuery(baseurl, query[, mode, language, …]) a class for preparing an query to an TAP service.
DALResults(votable[, url]) Results from a DAL query.
SIAResults(votable[, url]) The list of matching images resulting from an image (SIA) query.
SSAResults(votable[, url]) The list of matching images resulting from a spectrum (SSA) query.
SLAResults(votable[, url]) The list of matching spectral lines resulting from a spectal line catalog (SLA) query.
SCSResults(votable[, url]) The list of matching catalog records resulting from a catalog (SCS) query.
TAPResults(votable[, url]) The list of matching images resulting from an image (SIA) query.
Record(results, index) one record from a DAL query result.
SIARecord(results, index) a dictionary-like container for data in a record from the results of an image (SIA) search, describing an available image.
SSARecord(results, index) a dictionary-like container for data in a record from the results of an SSA query, describing an available spectrum.
SLARecord(results, index) a dictionary-like container for data in a record from the results of an spectral line (SLA) query, describing a spectral line transition.
SCSRecord(results, index) a dictionary-like container for data in a record from the results of an Cone Search (SCS) query, describing a matching source or observation.
AsyncTAPJob(url) This class represents a UWS TAP Job.
DALAccessError([reason, url]) a base class for failures while accessing a DAL service
DALProtocolError([reason, cause, url]) a base exception indicating that a DAL service responded with an error.
DALFormatError([cause, url, reason]) an exception indicating that a DAL response contains fatal format errors.
DALServiceError([reason, code, cause, url]) an exception indicating a failure communicating with a DAL service.
DALQueryError([reason, label, url]) an exception indicating an error by a working DAL service while processing a query.

Class Inheritance Diagram

Inheritance diagram of pyvo.dal.query.DALService, pyvo.dal.sia.SIAService, pyvo.dal.ssa.SSAService, pyvo.dal.sla.SLAService, pyvo.dal.scs.SCSService, pyvo.dal.tap.TAPService, pyvo.dal.query.DALQuery, pyvo.dal.sia.SIAQuery, pyvo.dal.ssa.SSAQuery, pyvo.dal.sla.SLAQuery, pyvo.dal.scs.SCSQuery, pyvo.dal.tap.TAPQuery, pyvo.dal.query.DALResults, pyvo.dal.sia.SIAResults, pyvo.dal.ssa.SSAResults, pyvo.dal.sla.SLAResults, pyvo.dal.scs.SCSResults, pyvo.dal.tap.TAPResults, pyvo.dal.query.Record, pyvo.dal.sia.SIARecord, pyvo.dal.ssa.SSARecord, pyvo.dal.sla.SLARecord, pyvo.dal.scs.SCSRecord, pyvo.dal.tap.AsyncTAPJob, pyvo.dal.exceptions.DALAccessError, pyvo.dal.exceptions.DALProtocolError, pyvo.dal.exceptions.DALFormatError, pyvo.dal.exceptions.DALServiceError, pyvo.dal.exceptions.DALQueryError

pyvo.dal.adhoc Module

Datalink classes and mixins

Classes

AdhocServiceResultsMixin(votable[, url]) Mixing for adhoc:service functionallity for results classes.
DatalinkResultsMixin(votable[, url]) Mixing for datalink functionallity for results classes.
DatalinkRecordMixin Mixin for record classes, providing functionallity for datalink.
DatalinkService(baseurl) a representation of a Datalink service
DatalinkQuery(baseurl[, id, responseformat]) a class for preparing an query to an Datalink service.
DatalinkResults(votable[, url]) The list of matching records resulting from an datalink query.
DatalinkRecord(results, index) a dictionary-like container for data in a record from the results of an datalink query,
SodaRecordMixin Mixin for soda functionallity for record classes.
SodaQuery(baseurl[, circle, range, polygon, …]) a class for preparing a query to a SODA Service.

Class Inheritance Diagram

Inheritance diagram of pyvo.dal.adhoc.AdhocServiceResultsMixin, pyvo.dal.adhoc.DatalinkResultsMixin, pyvo.dal.adhoc.DatalinkRecordMixin, pyvo.dal.adhoc.DatalinkService, pyvo.dal.adhoc.DatalinkQuery, pyvo.dal.adhoc.DatalinkResults, pyvo.dal.adhoc.DatalinkRecord, pyvo.dal.adhoc.SodaRecordMixin, pyvo.dal.adhoc.SodaQuery

pyvo.dal.exceptions Module

DAL Exceptions.

Classes

DALAccessError([reason, url]) a base class for failures while accessing a DAL service
DALProtocolError([reason, cause, url]) a base exception indicating that a DAL service responded with an error.
DALFormatError([cause, url, reason]) an exception indicating that a DAL response contains fatal format errors.
DALServiceError([reason, code, cause, url]) an exception indicating a failure communicating with a DAL service.
DALQueryError([reason, label, url]) an exception indicating an error by a working DAL service while processing a query.

Class Inheritance Diagram

Inheritance diagram of pyvo.dal.exceptions.DALAccessError, pyvo.dal.exceptions.DALProtocolError, pyvo.dal.exceptions.DALFormatError, pyvo.dal.exceptions.DALServiceError, pyvo.dal.exceptions.DALQueryError