STUDY ON INTEGRATION OF DATA MANAGEMENT ACTIVITIES BETWEEN WMO PROGRAMMES
(Submitted by the Secretariat)
Summary and Purpose of the Document The document contains the report on the study on integration of data management activities between WMO Programmes.
The meeting is invited to consider the report on the study on integration of data management activities between WMO Programmes, in particular the recommendations given in page iv to vi of its executive summary.
It has been recognised for many years that data and its effective management are essential for the success of scientific programmes. Data is the foundation on which our knowledge of the environment is built and data management is essential to ensure effective and efficient use of this resource. This is particularly true for international programmes, where data must be exchanged and understood regardless of language.
Most WMO Programmes have instituted various measures and activities to support their data management requirements. Although the aggregated requirements for each of the Programmes are unique, there are similarities and overlaps in many of their specific requirements. It is therefore likely that benefits could be gained from further coordination and integration of related data management activities among the Programmes.
The current state of the ensemble of WMO information systems is extremely complex. However, a review of the overall data management strategies for several Programmes reveals that most have several fundamental capabilities in common. Nearly all Programmes collect data, transmit these data to one or more processing centres, perform quality control, generate products, transmit these products to users, and archive the data and products for future use. Several Programmes further divide the data flow into real-time and delayed mode data streams as shown in the figure below.
Real-time and operational data flow is often essentially one-way, that is, products are routinely transmitted to collection centres or broadcast to users without any explicit action from the recipient. A variety of telecommunication services are used for real-time transmission including leased fixed circuits (both terrestrial and satellite) and the Internet. Delayed mode transmission utilizes the Internet, post and, less often, private circuits.
Access to data held in archives is usually provided on a two-way or request/reply basis. That is, users contact the archive, request products and these products are then transmitted to the user. Although the Internet is rarely used for real-time data transmission it is commonly used for access to archives.
Several Programmes utilize the standard protocols and data representation forms of the WMO Basic Systems for transmission of their real-time and operational delayed-mode data. These include the WWW, Marine Meteorology and Oceanographic Activities, the Aeronautical Meteorology programme, and the World Climate Programme. However, other Programmes utilize processing procedures that are entirely unique. Furthermore, even those Programmes that utilize the Basic Systems for operational needs have usually developed their own procedures to handle archives, research data, and ad hoc requests. For the most part these have been developed with little or no coordination between Programmes.
This lack of standardisation has led to a number of problems. Notably:
There is little connectivity between applications developed to serve the needs of different Programmes
There are a large number of different applications whose development has not been coordinated making integration of data sets technically challenging
Multidisciplinary application of meteorological, hydrological and oceanographic data is hampered by lack of agreed standards needed to effectively identify, acquire and use all of the relevant data
The multiplicity of systems operated for different Programmes has resulted in incompatibilities, duplication of effort and higher overall costs for Members. Continuing to develop systems in this uncoordinated manner will exacerbate these problems and will further isolate the WMO Programmes from each other and from the wider environmental community. It will increase the difficulty in sharing information between programmes, which is growing increasingly important for inter-disciplinary research.
This problem has been recognised within WMO for many years and a number of efforts have been made to address it. Beginning in the mid 1990s a series of inter-programme data management coordination meetings were held. Representatives of all of the scientific and technical programmes of WMO, as well as representatives of other international programmes with significant data management components were represented. These meetings provided a useful forum to exchange views and information on the activities of the various programmes and led to the effort to develop a comprehensive WMO information system, which could encompass and address the needs of all WMO Programmes.
WMO Congress, at its fourteenth session, agreed that an overarching approach to information management within WMO was required: a single coordinated global infrastructure, the Future WMO Information System (FWIS). FWIS would be used for the collection and sharing of information for all WMO and related international programmes. The FWIS vision provided a common roadmap to guide the evolution of the information system functions performed by current WMO Programmes into an integrated system that efficiently meets all of the requirements of Members for the relevant international environmental information.
Regarding data management standards, principles, practices and guidelines currently in place, for the most part, Programmes provide overall guidance on preferred practices for data management but do not specify binding standards. For Programmes that comprise sub-programmes, each of the sub-programmes has usually instituted its own data management practices and procedures with little guidance or direction from the parent Programme. A notable exception to this trend is the WWW, which has a long history of specifying comprehensive standards within the WMO Technical Regulations. The breadth and detail of data exchange standards and procedures specified for the WWW have certainly contributed to the success of the WWW over the past 40 years. Nonetheless, these regulations are perceived to be complex and inflexible by some of the other Programmes, which have consequently managed their own data management activities with more flexible and informal arrangements.
It is extremely difficult, if not impossible, to specify specific requirements for the diverse set of information needed to meet the requirements of all of the Programmes. Only a few Programmes have defined and documented most of their data management requirements. For the majority of Programmes, some requirements have been specified, usually for real-time data, while other requirements are only vaguely known.
By considering the requirements that have been articulated by one or more of the WMO Programmes or projects, the following emerging requirements can be identified.
A widely available and electronic (on-line) catalogue of all meteorological and related data for exchange to support WMO Programmes is required.
It should be possible to rapidly integrate real-time and non-real-time (archive) data sets to better interpret weather events in a climatological context.
There is a need to identify the potential of observation sites established by one Programme to meet the requirements of other Programmes.
Need to harmonize data formats, transmission standards, archiving and distribution mechanisms to better support inter-disciplinary use of data and products.
Require a standard method for station numbering beyond the existing WMO numbers, which is not adequate to define all GAW, climate, hydrological or agromet stations.
Need standard practices for the collection, electronic archival and exchange of metadata, both high-level and detailed, especially for stations and instruments.
The analysis of ongoing and planned activities of the WMO and related programmes indicates that there has been good cooperation and coordination between most Programmes. However, there is also considerable room for improvement. In general, shortcomings of existing data management activities within the WMO Programmes can be traced to one or more of three basic causes:
Insufficient recognition of the importance of data management in programme planning
For the most part, real-time collection and dissemination of operational data are well coordinated. Much of these data are carried over the GTS. However, even for these data many problems have been identified, principally loss of data and a lack of real-time monitoring information. Shortcomings in real-time data flow are usually due to a shortage of resources or current capabilities and are not due to a breakdown in coordination.
Collection and transmission of delayed mode data, use of data for by multiple programmes, and ad hoc access to data and products are problematic. These functions have not been as well coordinated, there has been more duplication of effort, and incompatible standards have been developed or used.
It is evident that many of the problems in existing data management systems and capabilities are primarily due to an historic lack of resources and are thus unlikely to be remedied by any recommendations made in this report. However, some deficiencies are primarily the result of poor communication or insufficient coordination and there is reason to hope that these problems could be remedied. Specifically:
There are a large number of different applications developed to serve the needs of different Programmes whose development has not been coordinated and there is little connectivity between applications resulting in inadequate/inflexible data exchange standards and incompatible data formats
Inflexible or incompatible data transmission standards, procedures and protocols have been adopted
Diverse and incomplete quality control of observational data and insufficient documentation of procedures for these data to be effectively used by other Programmes
No standard practices for the collection, electronic archival and exchange of metadata, both high-level and detailed, especially for stations and instruments
Lack of agreed standards needed to effectively identify, acquire and use all of the relevant data, particularly from archives, hampers multidisciplinary application of meteorological, hydrological and oceanographic data
Poor coordination between observing systems, making it difficult to identify the potential of observation sites established by one Programme to meet the requirements of other Programmes.
Within WMO, each Programme has traditionally worked independently to develop and implement its own plans and activities. Programmes have worked together towards a common goal only where possible benefits were clear and immediate. An example of success in this area can be found in the cooperation between the WWW and Programmes that use the Basic Systems to generate and transport their data and information.
There are many reasons for the traditional independence of the Commissions and Programmes.
First and foremost, coordination between departments and between Technical Commissions has historically not been a high priority within WMO. It is clear from the Technical Regulations that it is assumed most activities are to be carried out within a Commission. The WMO budget is defined and administered by Programmes and the departmental structure of the Secretariat mirrors that arrangement. Creation of inter-commission groups thus introduces significant complications in management, budgeting and reporting responsibilities.
Second, many experts and staff see coordination as a cost with little benefit. That is, the effort required to coordinate activities with others can significantly slow progress and may require compromises and introduce complexity that would not be needed if the activity were narrowly focused on the needs of a single Programme. Since major activities within WMO are usually initiated in response to a pressing need, time spent on coordination is often seen as introducing unacceptable delay. This attitude also applies to standards, which are developed only when there is a compelling need and, again, usually within a single Commission.
Third, bureaucratic rivalries between Commissions and departments are an unfortunate reality. This does not imply that directors or staff make deliberate decisions not to cooperate. Rivalry instead manifests itself in the feeling that “we know best” or “we can do it better”. Thus, each Programme sees its expertise as superior and its requirements unique. In such an atmosphere, it is quite rare for one Programme to approach another Programme or department to ask for assistance.
Over the past several years it has become increasingly evident that many scientific and technical problems are inherently interdisciplinary in nature. It has been recognized for more than a decade that data and information management is an area where better coordination and cooperation could lead to increased efficiency and improved services. New crosscutting WMO Programmes (Space and Natural Disaster Prevention and Mitigation) presage increased requirements for well-coordinated information management. Several attempts have been made to address this issue. However, the organizational and bureaucratic obstacles have made this difficult.
Although progress has been made over the past several years, current mechanisms for coordinating data management activities within and between departments in the Secretariat and between the Technical Commissions could be further improved. The principal problems within the Secretariat are the result of poor communication, budget issues and the low priority accorded to coordination. Between Technical Commissions the problems generally result from poor communication, complications resulting from reporting responsibilities, inappropriate expertise at meetings, competing or incompatible requirements, and the perception that joint projects are slow and unwieldy. The following recommendations are suggested to address these problems.
Within the Secretariat:
Establish a data management coordination team within the Secretariat. Members of this team, comprising representatives of all of the scientific and technical departments, would be responsible for ensuring the activities within their own department are coordinated with similar activities in other departments. The team should also develop and oversee plans for inter-programme activities to accomplish common goals. It would be worthwhile for the team to meet once every few months to discuss ongoing activities and accomplishments. However, recognizing that conflicting schedules would make it difficult to hold regular meetings, once the team were established it might be sufficient to exchange information via an e-mail mailing list.
Budget support for inter-commission work must be made explicit. A formal mechanism for funding cross-programme activities in data management should be defined. The preferred option is for Executive Council to establish, approve and commit funds to a crosscutting programme in data management. There are several models that could be used to manage this programme. It could be managed by dedicated staff within an existing department, an independent office could be created under the Deputy Secretary General, as was done with the WMO Space Programme, or it could be matrix-managed by staff from several departments, perhaps comprising the coordination team recommended above. If it is not possible to create a formal crosscutting programme then each department should be directed to establish a line within its budget to support this work.
Improve feedback from experts to the Secretariat. Experts attending inter-commission task teams as representatives of their Programme or Commission often do not provide sufficient feedback of the results to the responsible staff in the Secretariat (or their Commission). These experts should be required to submit reports on their participation, which should include the results of the meeting, with emphasis on the requirements and concerns of their particular Programme and if or how they have been addressed.
Improve circulation of meeting reports. By improving the dissemination of the most pertinent results of meetings, communication of important activities could be improved at little cost. All meeting reports should include a concise (less than one page) summary of their most important results. Concrete results, such as proposed standards, are particularly important. These summaries should be distributed to the directors and chiefs of all of the scientific and technical departments. All reports should be accessible via the Web and links to these reports should be prominent and obvious from the front page of each department’s Web site. It would probably be worthwhile to standardize the links to this information on each department’s home page.
Within and between Technical Commissions
Many issues affecting more than one Commission can be resolved using the current mechanism of “weakly inter-commission” teams. That is, those teams that are established within a Commission but with invited representatives of other relevant Commissions. From this point onward within this summary these will be referred to as “inter-programme teams”. To ensure the most effective experts are nominated, the attendance of the invited representatives should be funded by the Commission he/she represents. Since it is difficult to anticipate all opportunities for such inter-programme teams, every Programme should include some funds within its budget to support its participation in such efforts.
Some critical issues can best be addressed by “independent inter-commission” teams. In order to be most effective, these teams need to truly independent of any particular Commission. Membership of these teams should be based on the expertise needed with equal weight given to the requirements of all participating groups. Otherwise there is a perception that the views of one Commission (usually the designated lead) are accorded higher priority than the views of the others. If a crosscutting programme in data management were established (as suggested in recommendation b above), this programme would be responsible for the funding and management of such independent teams related to data management. If an umbrella programme is not established, then each inter-commission team should be directly funded by and report to Executive Council. Suggested mechanisms to manage such teams are discussed under recommendation b above. Since the participation in these teams will be supported by a dedicated budget, the team leader or responsible officer in the Secretariat should have a significant role in confirming team members.
Inter-agency, inter-organization coordination or data management activities can provide important benefits. Ocean data management, where JCOMM facilitates coordination with IOC, is a good example. Other Programmes within WMO that could benefit from further coordination with outside organizations include hydrology, agricultural meteorology and the World Climate and Weather Research Programmes. Officers responsible for these activities should reach out to other relevant organizations to investigate possible collaborative arrangements pertaining to data or information management.
Currently, FWIS is perceived as catering primarily to the requirements of CBS, while at the same time, moving too slowly to address pressing requirements of other Programmes. An independent inter-commission team, with representation from all of the players on an equal basis, should refocus further development and implementation of FWIS. The team should review the various components of FWIS and concentrate further efforts on a few specific and achievable objectives. Some users have suggested that highest priority should be given to developing an effective means to help users to find, acquire and use pertinent data from all WMO Programmes.
Continue to hold informal meetings of the Presidents of Technical Commissions. These meetings provide an effective forum for the presidents to exchange views and to keep abreast of significant developments and activities within the other Commissions. Reports of these meetings should be posted on the Web so that experts not in attendance might benefit from this same exchange of information.
Experts representing a Commission should be made explicitly aware of their responsibilities. Experts attending inter-programme or independent inter-commission task teams as representatives of their Programme or Commission should be reminded of their responsibilities in this capacity. Before attending the meeting they should be briefed of the Commission’s position on pertinent topics and should be given the authority to speak on the behalf of his/her president when necessary. Furthermore, effectiveness is enhanced when there is continuity in the experts chosen to attend follow-on meetings of the same group. These experts should submit reports on their participation to their president noting the requirements and concerns of their Commission and if or how they have been addressed by the meeting.