CYBERSKA

 A Cyberinfrastructure platform to meet the needs of data intensive radio astronomy on route to the SKA
Cyber SKA

Cyber SKA

Collaborators on the CANARIE Cyber SKA Project

Navigation

  • Public

CyberSKA CANARIE NEP History (2010-2011)

Last updated 1670 days ago by David Aikema

Canadian funding for CyberSKA is provided by CANARIE as part of their Network Enabled Platforms (NEP-2) program. This page will be used to keep track of the progress of the project. Note that some links on this page have restricted access.

Overview

The goal of this project is to create the beginnings of an custom-designed cyberinfrastructure for collaborative e-research with the Square Kilometre Array (SKA) and its pathfinders. The expected result is a working first version of the evolvable cyberinfrastructure that the SKA will require over its many forthcoming decades of project life to collect, store, analyze, re-use, and transform the unprecedented quantities of astrophysical data that will be collected by what will, in 2023, be the largest radio-telescope ever built.  The project interfaces emerging and future network platform technologies with leading-edge data intensive science in the form of major radio survey programs that pathfind SKA science and techniques.

Overview Documents

Project Reports

Technical Documents

Posters

Presentations/Demonstrations

Publications

  • M. Andrecut, S. S. Guram, S. J. George and A. R. Taylor. GALFACTS RFI Excision Methods. In Proceedings of the Astronomical Data Analysis Software and Systems Conference (ADASS XX), Boston, Massachusetts, Nov. 7-11, 2010.
  • S. S. Guram, M. Andrecut, S.J. George and A. R. Taylor. The GALFACTS Data Processing Pipeline. In Proceedings of the Astronomical Data Analysis Software and Systems Conference (ADASS XX), Boston, Massachusetts, Nov. 7-11, 2010.
  • V. Mahadevan and E. Rosolowsky. A Distributed Datacube Analysis Service for Radio Telescopes. In Proceedings of the Astronomical Data Analysis Software and Systems Conference (ADASS XX), Boston, Massachusetts, Nov. 7-11, 2010. 
  • P. Federl, A. Grimstrup, C. Kiddle and A. R. Taylor. On-line Access and Visualization of Multi-dimensional FITS Data. In Proceedings of the Astronomical Data Analysis Software and Systems Conference (ADASS XX), Boston, Massachusetts, Nov. 7-11, 2010.
  • C. Kiddle, M. Andrecut, A. Brazier, S. Chatterjee, E. Chen, J. Cordes, R. Curry, R. A. Este, O. Eymere, P. Federl, B. Fong, A. Grimstrup, S. Guram, V. Kaspi, R. Klodzinski, P. Lazarus, V. Mahadevan, A. Mourad, S. Mourad, P. Pragides, E. Rosolowsky, D. Said, A. Samoilov, C. Smith, I. Stairs, T. Tan, M. Tan, A. R. Taylor, and A. G. Willis. Looking Towards the Future of Radio Astronomy with the CyberSKA Collaborative Portal. In Proceedings of the Astronomical Data Analysis Software and Systems Conference (ADASS XX), Boston, Massachusetts, Nov. 7-11, 2010.
  • P. Federl, A. R. Taylor, A. Grimstrup, C. Kiddle, K. Robinson, M. Stephure and G. Yee. Remote Visualization of Large Multi-dimensional Radio Astronomy Data Sets. To appear in Proceedings of the Astronomical Data Analysis Software and Systems Conference (ADASS XXI), Paris, France, Nov. 6-10, 2011.
  • S. J. George, J. M. Stil, M. Andrecut and A. R. Taylor. Probing Magnetic Fields with GALFACTS. To appear in Proceedings of the Astronomical Data Analysis Software and Systems Conference (ADASS XXI), Paris, France, Nov. 6-10, 2011.
  • C. Kiddle, A. R. Taylor, J. Cordes, O. Eymere, V. Kaspi, D. Pigat, E. Rosolowsky, I. Stairs and A. G. Willis. CyberSKA: An On-line Collaborative Portal for Data-intensive Radio Astronomy. In Proceedings of the Gateway Computing Environments Workshop, Seattle, WA, Nov. 18, 2011.
  • M. Andrecut, J. M. Stil, and A. R. Taylor. Sparse Faraday Rotation Measure Synthesis. The Astronomical Journal, 143:33 (12pp), February 2012.

 

Milestones and Deliverables

Collaboration Services

  • SKA Canada / Cyber SKA Portal
    • Feb. 11, 2010 - SKA Canada Portal v0.6.0
      • initial portal adapted from the GeoChronos portal with just the base social networking functionality enabled
    • Mar. 15, 2010 - SKA Canada Portal v0.6.1
    • May 3, 2010 - SKA Canada Portal v0.8.0
      • upgraded to Elgg 1.7.1
      • updated event calendar plugin
      • added/customized task plugin
      • added publication plugin
      • other minor modifications/bug fixes
    • May 14, 2010 - SKA Canada / CyberSKA Portal v0.8.1
      • created a second CyberSKA Portal (www.cyberska.org)
        • SKA Canada exclusively for Canadian SKA Consortium related activities
        • CyberSKA Portal aimed more at cyberinfrastructure explored as part of this project and with other international collaborators
      • myGroups menu bar
      • editable group page layout
      • event calendar updates (new context menu, register/unregister)
      • invite any site member to a group (not just contacts)
      • task plugin updates (e-mail notification for tasks, task widget for groups)
      • language settings
      • administrator site wide e-mail
    • June 15, 2010 - SKA Canada / CyberSKA Portal v0.8.2
      • bug fixes,
      • full width theme
      • arxiv-widget added
      • inline PDF viewing (with Adobe Reader)
      • group e-mail functionality
    • July 13, 2010 - SKA Canada / CyberSKA Portal v0.8.3
      • bug fixes,
      • tag based file search
      • enhanced search functionality for publications
      • activity widget updates
      • task filtering
    • August 13, 2010 - SKA Canada / CyberSKA Portal v1.0.0
      • bug fixes
      • shared events between groups
      • addition of new Portal Support group
      • in-line viewing of text files
      • added folders plugin for organizing files into folders
      • MIME type recognition of FITS files, browsing of FITS files only
      • integration of FITS Viewer 0.2.0 for on-line visualization of FITS files
    • September 28, 2010 - SKA Canada / CyberSKA Portal v1.0.1
      • bug fixes
      • upgraded portal to Elgg 1.7.3
      • addition of member map plugin
      • support for animated gifs
      • recognition and in-line viewing of postscript files
      • forwarding of messages
      • integration of OAuth based third party application plugin (v0.6.0)
    • October 18, 2010 - SKA Canada / CyberSKA Portal v1.0.2
      • addition of RSS feed widget
      • integration of prototype distributed Data Management Service v0.6.0
    • November 4, 2010 - SKA Canada / CyberSKA Portal v1.0.3
      • enhanced registration features
      • plugin for inviting others to CyberSKA site (Tools->Invite)
      • better identification of e-mails sent by portal
      • users can now view resources (file, pages, etc.) they have access privileges for from a group dashboard, even if they are not a member (i.e. public resources)
      • active users widget
      • pause animation feature on member map
      • bug fixes
    • December 17, 2010 - SKA Canada / CyberSKA Portal v1.0.4
      • upgraded to Elgg 1.7.4
      • chat functionality added
      • updated Twitter plugin - works with Twitters new OAuth based authentication scheme
      • bug fixes
    • February 9, 2011 - SKA Canada / CyberSKA Portal v1.0.5
      • upgraded to Elgg 1.7.6
      • subgroup functionality added
      • replaced folder plug-in with File Tree plug-in - now supports hierarchical directories
      • upgraded to Group Custom Layout plug-in v1.3.3 (support for new widgets, improvements to color/background specification, other bug fixes and enhancements)
      • myDashboard and profile pages now have sidebar menus
      • notification e-mail sent to site approvers when someone registers for site
      • ammended wording in password reset messages to be less spam-like
      • increased limit on pages viewable in group pages widget
      • added delete button to file list view
      • added cancel button on task editing forms
      • improvements in file type recognition
      • some performance tuning (google maps code now loaded only on member map page)
      • disabled the chat module so we can explore some performance issues that are being encountered
    • April 4, 2011 - SKA Canada / CyberSKA Portal v1.2.0
      • users can now respond to site messages via e-mail directly; messages are still archived on the site as well (works best with plain text e-mails)
      • easier approach to sharing site resources with contacts (via sending message with a link)
      • page history now displays differences between revisions rather than the difference between that revision and the current version
      • default access levels for a group resource is now the group
      • default sub-page access levels are inherited from the parent page
      • delete button has been added to the list view for files
      • group member page has tabs like the main member page now
      • various bug fixes
    • May 19, 2011 - CyberSKA Portal v1.2.1
      • upgraded tinyMCE editor
      • added new Free HTML widget for dashboard and profile pages
      • added file access level in file view
      • a variety of bug fixes: various CSS/tinyMCE fixes, registration bug fixed, display proper icon for applications, edit event bug fixed, ...
    • June 30, 2011 - CyberSKA Portal v1.3.0
      • new site theme - new logo / banner/ colors/ home page layout - now features informational slide show
      • various site consistency issues were addressed
      • search by tag added for pages
      • comments added for events
      • various bug fixes
      • introduction of Distributed Data Management System v0.8.0 (all files migrated to new system)
      • new File widget (existing users of file widget should delete old one and add new one)
      • introduction of server side based FITS Viewer v0.8.0 (available via "View Movie" button for FITS files)
    • August 26, 2011 - CyberSKA Portal v1.3.1
      • can now send messages to multiple contacts and contact lists
      • repression of activity feed messages when resource modified more than once in short time period
      • chat functionality available again (via new plugin) - has since been disabled due to performance issues
      • added busy icon during data upload/submission process to notify user something is happening
      • various bug fixes
    • October 14, 2011 - CyberSKA Portal v1.3.2
      • groups can now have multiple administrators
      • various bug fixes
      • updated to Data Management Service v0.9.0 - support for FITS and measurement set (CASA) images as well as UVFITS and measurement set visibility data
      • thumbnails generated for FITS and CASA images, icon with text "UVW" for visibility data
      • mime types for existing portal file reevaluated with new headers extracted and thumbnails generated
      • in portal file module can select to view just "Image Data" or "Visibility Data" via side menu bar if desired
      • can now visualize both FITS and measurement set images uploaded to the portal using client side and server side image viewers

Applications

  • Applications Plugin
    • September 28, 2010 - Applications v0.6.0
      • Basic third party application plugin
      • Authentication via OAuth 1.0
      • Supports single sign-on
      • only admin can add applications
    • October 18, 2010 - Applications v0.6.1
      • read, write and execute permissions for applications
      • application information editable
      • enable members of a specific group to add applications (CyberSKA Developers)
      • bookmarking functionality to mark an application as a "group application"
      • tidying up of interface
    • April 4, 2011 - Applications v0.6.2
      • revamp of the application module user interface
      • added an application catalogue
      • added information page for each application
      • custom icon can now be associated with each application
      • addition of "My Applications" - bookmarking functionality like that for "Group Applications" to allow users to mark applications that are of interest to them
      • addition of a "Group Applications" widget for the group dashboard
      • various bug fixes
  • Imaging Applications
    • February 23, 2011 - CASA Pipeline - Initial Prototype
      • Web-based interface accessible via CyberSKA portal
      • mysql database for tracking jobs (in UWS format)
      • preliminary job manager for submitting jobs to Condor
    • August 26, 2011 - Application update
      • have added source counts application, visibility processing pipeline and GALFACTS pipeline third party application
  • PALFA Applications
    • August 20, 2010 - Candidate Viewer Prototype
      • connection to MSSQL server at Cornell
      • rating of candidates
      • user comments
      • form interface for searching / loading candidates for rating
      • saving / loading of form queries
    • August 20, 2010 - Diagnostics Tool Prototype
      • RFI Viewer - to examine quality of observed data
      • Observation Viewer - to show amount of data vs observation to date
      • Sky Map - to show area in sky that has been observed
    • December 2010 – Diagnostics Tool Update
      • Django server and application development
      • User interface for parameter input
      • Interactive tools for data tracking, RFI and SKY map
    • February 21, 2011 - Observation Scheduler
      • scheduler interface for Arecibo Observatory
      • integration of primary and secondary users
      • simplified interface
      • deployment to production server
    • February 21, 2011 - Top Candidates
      • Tool for browsing top ranked pulsar candidates
    • February 21, 2011 - Ratings
      • multi-threaded implementation
      • improved performance
      • recalculate outstanding ratings from UBC
      • implement a new architecture to store ratings
    • August 26, 2011 - PALFA third party application update
      • Discoveries App:
        • development has begun of a new app to provide up-to-date information about all PALFA discoveries and follow-up
      • ARCC Explorer (another candidate viewer - previously written by Kevin Stovall at the University of Texas at Brownsville)
        • integrated into CyberSKA via OAuth
        • this candidate viewer has been quite successful at training high-school students and undergrads how to find pulsars
      • PALFA Application Suite
        • the Candidate Viewer, Top Candidates, Observing Sign-Up and Diagnostics apps all received a makeover. They now have a consistent look and feel.
        • Candidate Viewer
          • New tool bar which provides user with plots and text information related to the candidate currently being viewed.
          • New query filters to allow to allow selection based on raw data type, observation type, and to omit previously classified candidates.
          • Added tool tips to many filters to improve the user experience.
          • Added/improved information about previously known pulsars near to the candidate being viewed.
          • Added hot-keys for nearly all possible actions.
        • The Observing Sign-Up
          • Sends notification emails to a PALFA mailing list.
          • Capable of recognizing small modifications in the telescope's schedule, and update information, but not spam PALFA group members.
        • Top Candidates
          • Supports added for multiple plots per candidate (to allow for further evidence for or against a candidate to be shared).
          • Added comment threads.
          • Support for superusers, who can highlight candidates to be re-observed.
          • Candidates can be added to a catalog widget, which can produce a telescope-ready listing of sources and coordinates, to facilitate re-observation.
        • Diagnostics
          • Custom diagnostic plots produced on demand, based on a series of filters entered by the user.
          • New types of diagnostics have been developed (still need to be integrated).
          • Existing diagnostics have been updated/improved (still need to be integrated).
    • Dec 19, 2011 - PALFA Third Party Applications Update
      • Discoveries App (developed at University of Texas at Brownville)
        • completed the application to replace static listing
        • keeps track of all new pulsar discoveries via a database
        • users can dynamically update the list and upload plots, files, etc for each discovery so follow-up observations can be kept track of
  • Interactive Application Service (IAS)
    • Jun 14, 2011 - Interactive Application Service v0.8.0
      • Prototype interactive application service deployed in CyberSKA test environment that is based on the GeoChronos Interactive Application Service
      • Deployed on top of ASPEN cloud computing platform at UofC
      • Enables user to dynamically launch and run common GUI based desktop applications in the cloud and access them remotely over the Web via VNC and ssh tunnelling service. Current applications supported include TOPCAT, Aladin, ds9, kviz and a CentOS desktop (i.e., virtual desktop support too)

Data Processing

  • GALFACTS Pipeline
    • August 20, 2010 - calmain (data calibration) and mapmain (gridding and mapping) programs updates
      • large parts of both programs rewritten to improve performance and reduce memory requirments
      • dependence on 3rd party libraries removed
      • new polynomial fitting functions
      • new RFI mitigation functions
    • February 20, 2011 - calmain and mapmain updates 
      • developed a new calibration routine in calmain to filter and compute the gain values for X, Y, and the phase variation phi vs channel, using Adaptive Total Variation Denoising - provides correction for Stokes parameters Qi and Ui
      • several polynomial fitting routines have been developed for mapmain using Tikhonov regularization and a. Gaussian elimination with pivoting,  b. LU factorization, c. Cholexky factorization, d. LDL' factorization, e. QR factorization
      • another faster and more stable polynomial fitting routing also developed for mapmain and used by default - Chebyshev polynomials and QR factorization with re-orthogonalization
      • basketweaving routing for mapmain has been rewritten to incorporate the new fitting methods; it has also been simplified and optimized - mapmain is now about 10 times faster than it was initially
      • quality of images produced has been significantly improved
    • August 26, 2011 - calmain and mapmain updates
      • some functions of calmain and mapmain rewritten to improve performance
      • new complex gain correction in calmain can be done by linear fitting, or by Gaussian kernel smoothing
      • current work on mapmain is focusing on Multi-beam clean and polarization leakage correction
      • a new beam model based on 2-dimensional Chebyshev polynomials has been developed as the core function for clean correction
      • a new Faraday Rotation Measure Synthesis method has been developed, and a paper is ready to be submitted for publication
      • the Faraday Rotation Measure Synthesis algorithm ahs been implemented and tested on synthetic data
      • new sets of GALFACTS images have been produced, reflecting the new changes in the pipeline
      • accessible/executable by third party application integrated with CyberSKA portal
  • PALFA Pipeline
    • August 20, 2010 - Update on PALFA v2.0 progress
      • unified qsub job managing / monitoring
      • discovery of new files for processing and adding to job pool
    • February 21, 2011 - Update on PALFA v2.0 progress
      • updated architecture of the pipeline
      • download module with pooling from Cornell
      • job pool module implementation for processing jobs
      • upload module to Cornell with diagnostics
    • August 26, 2011 - PALFA Pipeline Update
      • added support for upload results of single-pulse analysis to the Cornell-hosted results database
      • added utilities for performing common pipeline tasks, and monitoring performance, disk usage, etc.
      • improved documentation, error reporting and the overall robustness of the pipeline, making it even more automated
      • the pipeline was officially released to PALFA consortium members
      • deployed pipeline on CyberSKA hardware at UBC, McGill and also at CLUMEQ. The pipeline is being installed at the NRAO in Charlottesville, University of Texas at Brownsville, University of California at Berkeley, and Franklin and Marshall College
      • in total ~17000 data sets have been analysed with the new pipeline. This represents nearly 20% of all data sets analysed
      • PSR J1907+03 pulsar discovered. Identified several other top candidates.
  • PALFA Artificial Intelligence (AI)
    • July 2010 - Core AI system v0.1.0
      • Train AI by Samples
      • Make Decisions for new Record based on trained AI
      • Save / Load intermediate files for Decision Tree and Samples.
      • Test the whole and sub-decision of the AI.
    • August 2010 - PALFA AI GUI v0.1.0
      • 4 Features for Core AI system v0.1.0.
      • Load UBC / McGill data from database to intermediate file.
      • Generate random samples from intermediate Sample file.
      • Create / Modify data samples.
    • February 21, 2011 – PALFA AI GUI Update
      • data retrieval from multiple sources
      • data sampling and generation
      • model training and testing
      • AI documentation
    • August 26, 2011 - PALFA AI Update
      • AI algorithm has been improved and tested against historical data. High success rate is obtained for the data set. By applying the model to the latest data, we confirmed two pulsars classified recently
      • result analysis tool is integrated to the system. It is based on graphic rendering of probability distribution
      • candidate ID reporting is integrated into the system. This function provides access for users to extract the candidate IDs for further study and review
      • more testing is being done with sophisticated rating data. This set of data give further detailed description of pulsar and should be used to do the classification
      • we are doing data structuring and modelling in the database. More analysis of the data is under going. We expect to improve the results when more of the data is available
    • December 28, 2011 - PALFA AI Update
      • New algorithm based on probability density and statistics were developed within the AI system. The AI system can now be used not only for classifying the pulsar data with high accuracy, but also for describing the likelihood using a quantitative scale.
      • A database issue has been resolved and clarified. Duplicated and incomplete attributes in data have been analysed and dealt with using a proper algorithm.
      • Data retrieval and transportation problems have been resolved. New code has been developed to reduce the time needed for data communication and extraction.
      • A new method of data sampling for training and testing has been studied. The optimized datasethas been determined to generate the best result.
  • Cloud Computing Environment
    • July 26, 2011 - CyberSKA Cloud deployment
      • completed deployment of ASPEN cloud management software at UofC, UBCO, UBC and McGill

Visualization Services

  • FITS Viewer
    • August 13, 2010 - FITS Viewer v0.2.0
      • initial on-line prototype for visualization of FITS files uploaded using Elgg file plugin
      • JavaScript-based
      • interactive panning and zooming
      • histogram correction
      • color map adjustments
      • displays data values at pixels
    • September 28, 2010 - FITS Viewer v0.2.1
      • bug fixes
      • adaptive unit display based on range of values
      • RA--SIN, DEC-SIN and TAN projections are now supported
    • October 18, 2010 - FITS Viewer v0.2.2
      • bug fixes
      • FITS viewer accessible to public for files with public access
      • multiple coordinate systems supported (equatorial (B1950, J2000), galactic, ecliptic)
      • grid lines for different coordinate systems can be displayed
      • frame of multidimensional image to display can be selected
      • menu system
      • display of FITS header
      • save a screenshot to users computer
      • default histogram preset of 99.5%
    • November 4, 2010 - FITS Viewer v0.2.3
      • bug fixes
      • region statistics - user can now select a region on image and have basic statics in that region displayed (including: mean value of the pixel values, standard deviation about the mean, minimum value, maximum value, total number of pixels)
    • December 17, 2010 - FITS Viewer v0.4.0
      • grid drawing routines rewritten - bug fixes, grid clipped to inside of image
      • additions to region stats - sum, background removal stats, proper unit formatting
      • adjusted region stat calculations - interactive updates provided if they can be computed in less than 0.2 s, otherwise the computation is delayed until after the user releases the mouse button
      • added more optimization for c++ frame extraction on the server side
    • February 9, 2011 - FITS Viewer v0.4.1
      • implemented 2D gaussian fitting functionality (supports simulated annealing and  levenber/marquardt using forward finite differencing)
      • changing histogram/colormap now uses separate thread (web workers)
      • data that is visible in browser view is now processed first
      • added progress indicator while loading files
      • implemented permalink - a link that you can share with others that will load the image with the same settings - zoom, selection, histogram, colormap
      • screenshot can be saved to the CyberSKA (ELGG) file system
      • zooming is now possible with ctrol + left mouse button
      • added option to display pixel grid when zoomed in very far
      • added flashing logo with FITS Viewer version and link to CyberSKA
      • fixed layout issues with top menus/bottom status bar
      • fixed javascript cache regeneration bug (re-uploaded files can be viewed now)
      • fixed 'zoom to selection' so that it no longer clips
    • April 4, 2011 - FITS Viewer v0.6.0
      • added grid labeling
      • added thumbnail view showing entire image and location of current area zoomed into
        • dragging in thumbnail pans the image
        • clicking in thumbnail centers the clicked pixel on the screen
      • gaussian 2d fitting enhancements
        • redesigned UI
        • centre on centre of mass now works on images with negative values
        • more rendering options (show/subtract gauss values)
        • now stops automatically
      • added functionality to toggle Elgg decoration and maximize use of space in browser window
      • Elgg decoration toggle saved as a cookie (last choice is stored)
      • separate pop up windows for selecting colormap and histogram to maximize screen space for image
      • more colormap options
      • added permalink to screenshot upload
      • added total flux density calculation to region statistics
      • fixed permalink update in the URL so that it does not get saved in browser history (works in newest Chrome, Firefox, Safari, but not in Opera)
      • when new version of FITS Viewer is available, browser side caching is circumvented so that the new version should always load
      • implemented FITS loader which loads all data as AJAX
      • while loading FITS Viewer a thumbnail of the loading data is displayed
      • performance/memory/storage improvements
        • removed web worker implementation due to performance issues
        • store data as 2D array instead of 1D, allowing for larger FITS images
        • thumbnail calculation is now extraordinarily memory-efficient
        • optimized histogram/colormap calculation by caching
        • server side javascript cache computation much more memory efficient (at a slight expense in performance)
        • server now stores the javascript cache on disk compressed, resulting in less storage, faster loads, less server CPU (it used to compress this on the fly)
    • May 19, 2011 - FITS Viewer v0.6.6
      • zoom/unzoom functions in histogram window
      • FITS viewer should now be more patient with slower network speeds
      • added cancel button to loading window
      • Gaussian fit changes - random guess removed, added centre on extremum/min options, other minor changes
      • various bug fixes and cosmetic changes
    • June 30, 2011 - FITS Viewer v0.8.0 (Server side based FITS Viewer)
      • developed in collaboration with Calgary Scientific, using their PureWeb technology to provide Web-enabled access to a server side hosted visualization tool
      • supports interactive panning and zooming, histogram correction, color map adjustments, display pixel data value, intensity profiles, movie player, sharing of sessions
      • Flash, iPad and Android clients
      • known bugs in mapping of cursor location currently
    • Aug 26, 2011 - FITS Viewer v0.6.7 (Client side based FITS Viewer)
      • fits viewer now occupies all available vertical space by default
    • Nov 1, 2011 - FITS Viewer v0.8.1 (Server side based FITS Viewer)
      • added some new color maps (CubeHelix1, CubeHelix2) and color map RGB sliders
      • updated to newer version of PureWeb SDK
      • corrected bugs in mapping of cursor location

Data Management Services

  • Data Selector
    • July 13, 2010 - Data Access Interface v0.6.0
      • Initial prototype of data access tool for polarization cube FITS files
      • Released with preliminary GALFACTS C Field Data
      • Select region of interest, stokes parameters, channel range and width
      • uses static condor pool in background for generating FITS files for selection
    • August 13, 2010 - Data Access Interface v0.6.1
      • handle negative delta values for channel bandwidth
      • fixed tool tips so that they don't cause horizontal scrollbar to appear/disappear
      • bug fixes regarding calculating bandwidth values displayed in table of existing jobs
    • December 17, 2010 - Data Access Interface v0.6.2
      • GALFACTS D Field data, and N1 Run data added
      • added some minor optimization to data extraction code on server side
      • minor bug fixes
  • Data Management Service
    • August 20, 2010 - Data Management and Query Service v0.5.0
      • distributed iRODS repositories at UBCO and UofC
      • PostgreSQL database for image metadata (IVOA metadata standards applied)
      • RESTful API for query services (spatial, temporal and spectral queries supported)
      • Mosaicing of multi-dimensional images
      • Download of data
    • Oct 18, 2010 - Data Management and Query Service v0.6.0
      • Integration of data management service with Elgg File plugin (each file appears just as though it was a local file)
      • iRODS repository treated as separate type of filestore object - interaction via RESTful API - read only access at moment
      • Elgg object created (static/manual process currently) for each file in iRODS repository
      • FITS file headers pulled from RESTful API
      • support for in-line FITS visualization (remote file downloaded to portal site and javascript cache file generated if not already present)
      • iRODS data stores at UBCO and UofC
      • when downloading file, redirected to UBCO server
      • PostgreSQL database for image (FITS) metadata (adherent to VO metadata standards)
      • Query service with RESTful API (spatial, temporal and spectral queries supported) integrated as third party applications
      • Query service also supports simple data workflows that can carry out segmentation, mosaicing, plane extraction, compression and staging of images returned by the query
    • June 30, 2011 - Distributed Data Management Service v0.8.0
      • enhancement and full integration of iRODS based distributed data management system with production portal (all files converted over to new system)
      • asynchronous generation of FITS cache files when uploaded
      • 2 legged OAuth between portal and data management service
      • addition of Advanced Upload Tool - uses JUpload v5.0.2 java applet
        • ftp used for uploads
        • multiple file uploads supported
        • allows for uploads of files > 200MB
      • addition of Advanced Download Tool - uses Java Webstart version of CADC download manager
        • http used for downloads
      • deployed UofC data management server - more deployments to come soon
    • October 14, 2011 - Distributed Data Management Service v0.9.0
      • now supports mime type recognition for FITS and measurement set (CASA) images as well as UVFITS and measurement set visibility data
      • measurement sets can be uploaded as .tar or .tar.gz files
      • header extraction for all four data types supported and is displayed in file view on portal
      • thumbnails generated for FITS and CASA images
    • November 2, 2011 - Distributed Data Management Service v0.9.1
      • running at two sites (UofC, UBCO)
      • set up replication of files (
      • problems with threaded file transfers at the moment
    • December 20, 2011 - Distributed Data Management Service v0.9.2
      • updated version of header extractor - compression of some of the fields
      • minor bug fixes
  • GALFACTS Database
    • August 20, 2010 - Database Model
      • database model for storing GALFACTS raw data and calibration data
      • aimed at reducing storage requirements and adding flexibility in accessing data
    • February 22, 2011 - Database Implementation
      • implemented a database management system (DBMS) for the GALFACTS pipeline project using DB2
      • aimed at reducing storage required for GALFACTS in addition to providing the required infrastructre to visualize raw data
      • tested with a small sample dataset
    • December 31, 2011 - Database Update
      • Investigate methods for automatic detection of faults in GALFACTS surveys
      • Propose new methods for automatic detection of faults in GALFACTS surveys
Powered by Elgg