OMB guidelines on information quality

OFFICE OF MANAGEMENT AND BUDGET Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies AGENCY: Office of Management and Budget, Executive Office of the President. ACTION: Final Guidelines, with Request for Comments ________________________________________________________________________ SUMMARY: These guidelines implement section 515 of the Treasury and General Government Appropriations Act for Fiscal Year 2001 (Public Law 106-554). Section 515 directs the Office of Management and Budget (OMB) to issue government-wide guidelines that Aprovide policy and procedural guidance to Federal agencies for ensuring and maximizing the quality, objectivity, utility, and integrity of information (including statistical information) disseminated by Federal agencies.@ Within one year after OMB issues these guidelines, agencies must issue their own implementing guidelines that include Aadministrative mechanisms allowing affected persons to seek and obtain correction of information maintained and disseminated by the agency@ that does not comply with the OMB guidelines. OMB is also requesting additional comment for 30 days on the Acapable of being substantially reproduced@ standard (paragraphs V.3.B, V.9, and V.10) which is issued on an interim final basis. DATES: Effective Date: October 1, 2001. Comment Date: Comments on the Acapable of being substantially reproduced@ standard in paragraphs V.3.B, V.9, and V.10 must be submitted by [INSERT DATE 30 DAYS FROM DATE OF PUBLICATION IN THE FEDERAL REGISTER]. ADDRESSES: Please submitted comments to Brooke J. Dickson of the Office of Information and Regulatory Affairs, Office of Management and Budget, Washington, DC 20503. Comments can also be e-mailed to informationquality@omb.eop.gov. FOR FURTHER INFORMATION CONTACT: Brooke J. Dickson, Office of Information and Regulatory Affairs, Office of Management and Budget, Washington, DC 20503. Telephone (202) 395-3785. SUPPLEMENTARY INFORMATION: In section 515(a) of the Treasury and General Government Appropriations Act for Fiscal Year 2001 (Public Law 106-554; H.R. 5658), Congress directed the Office of Management and Budget (OMB) to issue, by September 30, 2001, government-wide guidelines that Aprovide policy and procedural guidance to Federal agencies for ensuring and maximizing the quality, objectivity, utility, and integrity of information (including statistical information) disseminated by Federal agencies... .@ Section 515(b) goes on to state that the OMB guidelines shall: A(1) apply to the sharing by Federal agencies of, and access to, information disseminated by Federal agencies; and A(2) require that each Federal agency to which the guidelines applyB A(A) issue guidelines ensuring and maximizing the quality, objectivity, utility, and integrity of information (including statistical information) disseminated by the agency, by not later than 1 year after the date of issuance of the guidelines under subsection (a); A(B) establish administrative mechanisms allowing affected persons to seek and obtain correction of information maintained and disseminated by the agency that does not comply with the guidelines issued under subsection (a); and A(C) report periodically to the DirectorB A(i) the number and nature of complaints received by the agency regarding the accuracy of information disseminated by the agency; and A(ii) how such complaints were handled by the agency.@ These guidelines are to be issued Aunder sections 3504(d)(1) and 3516" of the Paperwork Reduction Act of 1995; pursuant to section 3503 of that Act, the authorities of the OMB Director are carried out by the Administrator of the Office of Information and Regulatory Affairs. Background The focus of section 515 is on the Federal Government=s information dissemination activities. Indeed, Federal agencies have disseminated information to the public for decades. Until recently, agencies have disseminated information principally by making paper copies of documents available to the public. In recent years, however, Federal information dissemination has grown due to the advent of the Internet, which has ushered in a revolution in communications. The Internet has enabled Federal agencies to disseminate an ever-increasing amount of information. Congress has strongly encouraged the Executive Branch=s dissemination efforts in statutes that include particular dissemination activities and in the government-wide dissemination provisions of the Paperwork Reduction Act of 1995 (44 U.S.C. chapter 35) (the PRA). In addition, the Executive Branch=s strong support for information dissemination is reflected in the dissemination provisions of OMB Circular A-130, AManagement of Federal Information Resources,@ as well as in the provisions in OMB Circular A-110, AUniform Administrative Requirements for Grants and Agreements With Institutions of Higher Education, Hospitals, and Other Non-Profit Organizations,@ related to a Freedom of Information Act request for research data relating to published research findings produced under an award that were used by the Federal Government in developing an agency action that has the force and effect of law (64 FR 54926; October 8, 1999). Section 515 builds upon the existing agency responsibility to ensure information quality. According to the PRA, agency Chief Information Officers (CIOs) must manage information resources to Aimprove the integrity, quality, and utility of information to all users within and outside the agency, including capabilities for ensuring dissemination of public information, public access to government information, and protections for privacy and security.@ Before an agency collects information from 10 or more persons, the agency must seek public comment Ato enhance the quality, utility, and clarity of the information to be collected.@ The agency then must obtain OMB approval that is based upon an evaluation of the agency=s need for the information, the Apractical utility@ of the information to be collected, and the minimization of burden that would be imposed on the public in responding to the collection. The CIO must certify to OMB that the agency, Ato the maximum extent practicable, uses information technology to reduce burden and improve data quality.@ In developing these guidelines to implement section 515, OMB recognized that Federal agencies disseminate many types of information in many different ways. A few examples can only begin to describe the breadth of information disseminated by the Federal government. Agencies disseminate statistical information, such as the aggregated information from the 2000 Census and the monthly and quarterly economic reports issued by the Bureau of Economic Analysis and the Bureau of Labor Statistics. Agencies disseminate information that aids members of the public in their daily activities, such as the National Weather Service=s weather reports and the FAA=s air travel advisories. Agencies disseminate information about health, safety, and environmental risks and information that they collect from regulated entities, such as EPA=s dissemination of Toxic Release Inventory information. Agencies also disseminate technical information that they create or obtain in the course of developing regulations, often involving scientific, engineering, and economic analysis. Agencies disseminate information when they issue reports and studies. Moreover, agencies provide the public with basic descriptions of agency authorities, activities and programs, along with the contact information for the public to interact with and access that information or those services. Underlying Principles In accordance with section 515, OMB has designed the guidelines to help agencies ensure and maximize the quality, utility, objectivity and integrity of the information that they disseminate (meaning to share with, or give access to, the public). It is crucial that information Federal agencies disseminate meets these guidelines. In this respect, the fact that the Internet enables agencies to communicate information quickly and easily to a wide audience not only offers great benefits to society, but also increases the potential harm that can result from the dissemination of information that does not meet basic information quality guidelines. Recognizing the wide variety of information Federal agencies disseminate and the wide variety of dissemination practices that agencies have, OMB developed the guidelines with several principles in mind. First, OMB designed the guidelines to apply to a wide variety of government information dissemination activities that may range in importance and scope. OMB also designed the guidelines to be generic enough to fit all media, be they printed, electronic, or in other form. OMB sought to avoid the problems that would be inherent in developing detailed, prescriptive, Aone-size-fits-all@ government-wide guidelines that would artificially require different types of dissemination activities to be treated in the same manner. Through this flexibility, each agency will be able to incorporate the requirements of these OMB guidelines into the agency=s own information resource management and administrative practices. Second, OMB designed the guidelines so that agencies will meet basic information quality standards. Given the administrative mechanisms required by section 515 as well as the standards set forth in the PRA, it is clear that agencies should not disseminate substantive information that does not meet a basic level of quality. We recognize that some government information may need to meet higher or more specific information quality standards than those that would apply to other types of government information. The more important the information, the higher the quality standards to which it should be held, for example, in those situations involving Ainfluential scientific or statistical information@ (a phrased defined in these guidelines). The guidelines recognize, however, that information quality comes at a cost. Accordingly, the agencies should weigh the costs (for example, including costs attributable to agency processing effort, respondent burden, maintenance of needed privacy, and assurances of suitable confidentiality) and the benefits of higher information quality in the development of information, and the level of quality to which the information disseminated will be held. More specifically, the OMB guidelines state that Aagencies shall have a basic standard of quality (including objectivity, utility, and integrity) as a performance goal ...@. We note, in the scientific context, that in 1996 the Congress, for health decisions under the Safe Drinking Water Act, has already adopted a basic standard of quality for the use of science in agency decisionmaking. Under 42 U.S.C. 300g-1(b)(3)(A), an agency is directed, Ato the degree that an Agency action is based on science,@ to use A(i) the best available, peer-reviewed science and supporting studies conducted in accordance with sound and objective scientific practices; and (ii) data collected by accepted methods or best available methods (if the reliability of the method and the nature of the decision justifies use of the data).@ We also note that the OMB guidelines call for an additional level of quality Ain those situations involving influential scientific or statistical information.@ The additional level of quality concerns a standard of care for scientific or statistical analytical results, a Acapable of being substantially reproduced@ standard that is discussed below. We further note that in the 1996 amendments to the Safe Drinking Water Act the Congress adopted a basic quality standard for the dissemination of public information about risks of adverse health effects. Under 42 U.S.C. 300g-1(b)(3)(B), the agency is directed, Ato ensure that the presentation of information [risk] effects is comprehensive, informative, and understandable.@ The agency is further directed, Ain a document made available to the public in support of a regulation [to] specify, to the extent practicable B (i) each population addressed by any estimate [of applicable risk effects]; (ii) the expected risk or central estimate of risk for the specific populations [affected]; (iii) each appropriate upper-bound or lower-bound estimate of risk; (iv) each significant uncertainty identified in the process of the assessment of [risk] effects and the studies that would assist in resolving the uncertainty; and (v) peer-reviewed studies known to the [agency] that support, are directly relevant to, or fail to support any estimate of [risk] effects and the methodology used to reconcile inconsistencies in the scientific data.@ We urge each agency in developing its guidelines to evaluate whether adopting or adapting these basic Congressional standards would be appropriate for judging the quality of disseminated scientific or statistical information. Third, OMB designed the proposed guidelines so that agencies can apply them in a common-sense and workable manner. It is important that these guidelines do not impose unnecessary administrative burdens that would inhibit agencies from continuing to take advantage of the Internet and other technologies to disseminate information that can be of great benefit and value to the public. In this regard, OMB encourages agencies to incorporate the standards and procedures required by these guidelines into their existing information resources management and administrative practices rather than create new and potentially duplicative or contradictory processes. The primary example of this is that the guidelines recognize that, in accordance with OMB Circular A-130, agencies already have in place well-established information quality standards and administrative mechanisms that allow persons to seek and obtain correction of information that is maintained and disseminated by the agency. Under the OMB guidelines, agencies need only ensure that their own guidelines are consistent with these OMB guidelines, and then ensure that their administrative mechanisms satisfy the standards and procedural requirements in the new agency guidelines. Similarly, agencies may rely on their implementation of the Federal Government=s computer security laws (formerly, the Computer Security Act, and now the computer security provisions of the PRA) to establish appropriate security safeguards for ensuring the Aintegrity@ of the information that the agencies disseminate. Summary of OMB Guidelines These guidelines apply to Federal agencies subject to the Paperwork Reduction Act (44 U.S.C. chapter 35). Agencies are directed to develop information resources management procedures for reviewing and substantiating (by documentation or other means selected by the agency) the quality (including the objectivity, utility, and integrity) of information before it is disseminated. In addition, agencies are to establish administrative mechanisms allowing affected persons to seek and obtain, where appropriate, correction of information disseminated by the agency that does not comply with the OMB or agency guidelines. Consistent with the underlying principles described above, these guidelines stress the importance of having agencies apply these standards and develop their administrative mechanisms so they can be implemented in a common sense and workable manner. Moreover, agencies must apply these standards flexibly, and in a manner appropriate to the nature and timeliness of the information to be disseminated, and incorporate them into existing agency information resources management and administrative practices. Section 515 denotes four substantive terms regarding information disseminated by Federal agencies: quality, utility, objectivity, and integrity. It is not always clear how each substantive term relates -- or how the four terms in aggregate relate -- to the widely divergent types of information that agencies disseminate. The guidelines provide definitions that attempt to establish a clear meaning so that both the agency and the public can readily judge whether a particular type of information to be disseminated does or does not meet these attributes. In the guidelines, OMB defines Aquality@ as the encompassing term, of which Autility,@ Aobjectivity,@ and Aintegrity@ are the constituents. AUtility@ refers to the usefulness of the information to the intended users. AObjectivity@ focuses on whether the disseminated information is being presented in an accurate, clear, complete, and unbiased manner, and as a matter of substance, is accurate, reliable, and unbiased. AIntegrity@ refers to security -- the protection of information from unauthorized access or revision, to ensure that the information is not compromised through corruption or falsification. OMB modeled the definitions of Ainformation,@ Agovernment information,@ Ainformation dissemination product,@ and Adissemination@ on the longstanding definitions of those terms in OMB Circular A-130, but tailored them to fit into the context of these guidelines. In addition, agencies have two reporting requirements. The first report, implemented no later than one year after the issuance of these OMB guidelines (no later than October 1, 2002), must provide the agency=s information quality guidelines that describe administrative mechanisms allowing affected persons to seek and obtain, where appropriate, correction of disseminated information that does not comply with the OMB and agency guidelines. The second report is an annual fiscal year report to OMB (to be first submitted on January 1, 2004) providing information (both quantitative and qualitative, where appropriate) on the number, nature, and resolution of complaints received by the agency regarding its perceived or confirmed failure to comply with these OMB and agency guidelines. Public Comments and OMB Response Section 515(a) required OMB to provide the public and the Federal agencies the opportunity to comment on these guidelines. OMB worked with Federal agencies, through a working group and through an inter-agency comment process, in the development of the proposed guidelines. The proposed guidelines were published in the Federal Register on June 28, 2001 (66 Fed. Reg. 34489) providing a public comment period of 45 days. OMB received a total of 100 comments from academic institutions (36), Federal agencies (26), individual members of the public (7), associations affiliated with academia (5), associations affiliated with medical, social science or science interests (15), associations affiliated with Federal Government interests (4), and associations affiliated with industry interests (7). General Concerns. Many comments expressed support for the idea of government-wide quality standards for information disseminated by Federal agencies. Comments also expressed support for OMB=s commitment to creating flexible general guidelines and to minimizing the administrative costs and burdens that these guidelines will impose. The majority of comments focused on two aspects of the proposed guidelines: suggestions for placing limitations on the administrative correction mechanisms requirements of the statute; and the need to clarify specific definitions and other terms found in the guidelines. Many comments raised questions and concerns about how these guidelines interact with existing statutes and policies, including the Paperwork Reduction Act and the Government Performance and Results Act. We have attempted to draft these guidelines in a way that addresses the requirements of section 515, but does not impose a completely new and untried set of standards upon Federal agencies. We encourage agencies to consider the effect of relevant existing statutes and policies in the development of their own guidelines. Administrative Mechanisms. These guidelines require agencies to establish administrative mechanisms allowing affected persons to seek and obtain, where appropriate, correction of information maintained and disseminated by the agency that does not comply with the OMB guidelines. Many comments suggested that limits be imposed on the types of information that should be subject to these guidelines, in particular, information that is disseminated by agency libraries. OMB agrees that archival information disseminated by Federal agency libraries (for example, Internet distribution of published articles) should not be covered by these guidelines, given that libraries do not endorse the information that they disseminate. Moreover, an agency=s dissemination of public filings (for example, corporate filings with the Securities and Exchange Commission) is not covered by these guidelines. In each of these situations, the agencies have not authored these documents and have not adopted them as representing the agencies= views. By disseminating these materials, the agencies are simply ensuring that the public can have quicker and easier access to materials that are publicly available. In developing its implementing guidelines, and in accordance with the criteria set forth in these guidelines, each agency should evaluate and identify the types of information that it disseminates that will be subject to its guidelines. In addition, comments also raised the concern that the guidelines would apply to Apreliminary@ information, and they recommended that the guidelines exclude such information. OMB appreciates the concerns that these comments have raised. However, OMB does not believe that an exclusion for Apreliminary@ information is necessary or appropriate. It is still important that the quality of preliminary information be ensured and that preliminary information be subject to the administrative complaint-and-correction process. A few comments stated that affected information should be limited to information used in agency rulemaking. While this has been the position of previous policies which these guidelines are not intended to modify or replace (see, e.g., section ___.36(d) in OMB Circular A-110), we believe the plain meaning and intent of section 515 covers the larger government information universe. Based on the public comments received, these guidelines allow agencies to determine the appropriate level of correction for a complaint received. Several comments suggested that agencies use disclaimers to distinguish the status of information, a practice that agencies should consider adopting as they consider their information holdings. OMB received detailed discussion on the requirement that agencies develop administrative mechanisms allowing for affected persons to Aseek and obtain correction of information that does not comply with OMB=s guidelines.@ Members of the scientific community expressed strong concerns about the possibility of a Federal agency that would Acorrect@ scientific information without carrying out the scientific analysis to support the correction. Comments from all fields suggested in various ways that challenging individuals should be Arequired to openly state his/her relationship with the data/information (familiarity/expertise) and provide information [as] to his/her interest in it.@ Comments also pointed out great potential for abuse of this process. As one association summarized, AThis could be seen to provide grounds for interested parties to demand access to underlying data, to compel the government to replicate research findings (at great expense and with unnecessary delay), or in other ways impede, discredit, harass or stymie research.@ For example, the National Oceanic and Atmospheric Administration (NOAA) explained that they receive numerous complaints from the public when they miss a weather forecast. ADoes this mean that the NWS [National Weather Service] could be requested to change a forecast after the fact? Or could someone with an economic interest challenge official observational data which could affect the value of an insurance payment?@ asks NOAA. Overall, OMB does not envision administrative mechanisms that would burden agencies with frivolous claims. Instead, the correction process should serve to address the genuine and valid needs of the agency and its constituents without disrupting agency processes. Agencies, in making their determination of whether or not to correct information, may reject claims made in bad faith or without justification, and are required to undertake only the degree of correction that they conclude is appropriate for the nature and timeliness of the information involved, and explain such practices in their annual fiscal year reports to OMB. Numerous comments provided language to clarify or limit the term, Aaffected persons.@ One academic institution suggested that the term, Aaffected persons,@ reflects a criterion of Adirect measurable impact with significant personal consequence.@ Other academic institutions suggested that Aaffected persons should not be permitted to challenge the substance of information without showing that a qualified scientist has found fault with its quality or integrity.@ Similarly, some comments argued that the ability to correct scientific information should be limited only to other scientists. Several associations suggested that OMB identify the types of information that could be challenged rather than to focus on the characteristics of a Alegitimate@ challenger. OMB considered these comments at length. Our conclusion is that Aaffected persons@ are people who may benefit or be harmed by the disseminated information. This includes persons who are seeking to address information about themselves as well as persons who use information. However, each agency should consider how persons (which includes groups, organizations and corporations, as defined by the Paperwork Reduction Act) will be affected by the agency=s information. Agencies should address the issue of Aaffected persons@ in consultation with their constituents through the public comment process that agencies will provide after drafting their proposed guidelines and before submitting them for OMB review. These guidelines require that an agency official be designated to receive and resolve complaints regarding information that does not comply with either the OMB guidelines or the agency=s guidelines. In the proposed guidelines, we required, with a limited exception, that the Chief Information Officer (CIO) of the agency have this responsibility. Of the government agencies that commented on this provision, many pointed to their specific agency practices on information quality and their designation of a Aquality official@ who was not necessarily working under the agency CIO. Recognizing that some agencies may have specific officials in place to address quality issues, the final guidelines allow agencies to designate an appropriate official. Agencies may also designate multiple officials, i.e., based on the needs of individual agency components, as long as there is a single official with these overall responsibilities designated at the agency level. The authorized official also needs to consult with the CIO on quality matters pertaining to information disseminated by the agency. Agencies need to respond to complaints in a manner appropriate to the nature and extent of the complaint. Examples of appropriate responses, as suggested by comments, include personal contacts via letter or telephone, form letters, press releases or mass mailings that correct a widely disseminated error or address a frequently raised complaint. Agencies may want to utilize other methods of response under existing agency practices. For example, for agencies with a high volume of complaints, it is acceptable for the agency to describe a sample of those complaints in the annual fiscal year report to OMB. For categories of inconsequential or trivial complaints identified in the agency guidelines, an agency may decide that no response is necessary. Agencies should describe to OMB as part of the annual fiscal year report the chosen response mechanisms and how they are working. Definitions and Other Terms. Section 515 denotes four substantive terms regarding information disseminated by Federal agencies: quality, utility, objectivity, and integrity. We have defined Aquality@ as an encompassing term. The organizations and individuals that submitted comments did not object to having Aquality@ defined as an encompassing term, but suggested that we should discuss each term separately. The principles laid out in the proposed guidelines, stated one comment, create Asubjective definitions@ of the four terms. This comment warned OMB that Asubjective definitions of quality, objectivity, utility, and integrity could cause agencies to delay data release or disregard data for fear of challenge.@ Other comments expressed similar views, or as one association observed, AScience does not recognize a sliding scale of quality.@ These guidelines reflect OMB=s determination that Aquality,@ Autility,@ Aobjectivity,@ and Aintegrity@ are closely interrelated concepts in the context of these guidelines. Collectively, these terms address the following three aspects of the information that is to be disseminated: whether the information is useful to the intended users of the information; whether the disseminated information is being presented in an accurate, clear, complete, and unbiased manner in both presentation and as a characteristic that should be inherent to quality information; and whether the information has been protected from unauthorized access or revision. Some comments stated that OMB was Aexceeding the statutory mandate@ and going beyond ACongressional intent@ in specifying scientific and statistical information in these guidelines. Others felt that we should simply acknowledge that the scientific and statistical communities already have practices and standards for their information, rather than create another set of standards for these information types. OMB does not agree with those comments that said the proposed guidelines went beyond the statute in covering statistical and scientific information. Section 515 expressly states that its scope includes statistical information. Moreover, section 515 has no exclusion for scientific information, and in many respects it is very similar to (and overlaps with) statistical information. OMB, however, does appreciate the concerns that the comments raised about the guidelines not creating another set of standards for statistical and scientific information. Our guidelines do not seek to impose new standards on these communities, but to reiterate the standards that are already held in those communities. Recognizing public interest in medical and public health information, we have specifically added a provision stating, AAgencies shall adopt specific standards of quality that are appropriate for the various categories of information they disseminate.@ For example, OMB encourages agencies, in crafting their agency-specific guidelines, to promote objectivity in information quality in ways that protect the confidentiality of research subjects and encourage public participation in research. These concerns are particularly salient in medical and public health research. A number of comments regarded our discussion of ensuring that scientific information be Asubstantially reproducible@ as requiring agencies to replicate original data and to perform independent analysis upon all scientific information disseminated by the agency. We have responded to these concerns in a number of ways. First, we make it clear that what we now refer to as the Acapable of being substantially reproduced@ standard applies to analytical results that are disseminated, and does not apply to the original or supporting data. Thus, replication of original data is not required. Second, the Acapable of being substantially reproduced@ standard is applicable only to Ainfluential@ scientific and statistical information as defined in the guidelines. Third, the guidelines call for the agency to determine that Ainfluential@ analytical results be capable of being substantially reproducible by independent analysis. We intend this standard to say that, if appropriately qualified persons used the same or a similar methodology, they would be expected to achieve similar findings and results. Based on the concerns expressed in the comments, we expanded upon our discussion of Acapable of being substantially reproduced@ in our definition of Aobjective,@ and added two explanatory definitions. We state, in paragraphs V.3.B, V.9, and V.10: In addition, Aobjectivity@ involves a focus on ensuring accurate, reliable, and unbiased information. In a scientific or statistical context, the original or supporting data shall be generated, and the analytical results shall be developed, using sound statistical and research methods. i. If the results have been subject to formal, independent, external peer review, the information can generally be considered of acceptable objectivity. ii. In those situations involving influential scientific or statistical information, the results must be capable of being substantially reproduced, if the original or supporting data are independently analyzed using the same models. Reproducibility does not mean that the original or supporting data have to be capable of being replicated through new experiments, samples or tests. iii. Making the data and models publicly available will assist in determining whether analytical results are capable of being substantially reproduced. However, these guidelines do not alter the otherwise applicable standards and procedures for determining when and how information is disclosed. Thus, the objectivity standard does not override other compelling interests, such as privacy, trade secret, and other confidentiality protections. AInfluential@ when used in the phrase Ainfluential scientific or statistical information@ means the agency expects that information in the form of analytical results will likely have an important effect on the development of domestic or international government or private sector policies or will likely have important consequences for specific technologies, substances, products or firms. ACapable of being substantially reproduced@ means that independent reanalysis of the original or supporting data using the same methods would generate similar analytical results, subject to an acceptable degree of imprecision. As a general matter, in the scientific and research context, we regard technical information that has been subjected to formal, independent, external peer review as presumptively objective. An example of a formal independent external peer review is the review process used by scientific journals. However, depending on the nature and timeliness of the information involved, an agency may decide that peer review is not necessary or appropriate. On the other hand, in those situations involving influential scientific or statistical information, the substantial reproducibility standard is added as a quality standard above and beyond some peer review quality standards. In the definition of Ainfluential,@ when used in the phrase Ainfluential scientific or statistical information,@ we note that the manner in which people perceive the scientific or statistical information can have important consequences for specific policies, technologies, substances, products, and firms. Based on concerns with the Asubstantially reproducible@ standard, a number of comments suggested that OMB should repropose this standard for additional public comment, rather than going final at this time. While, in deference to the statutory deadline, OMB is issuing the Acapable of being substantially reproduced@ standard (paragraphs V.3.B, V.9, and V.10), OMB is doing so on an interim final basis. We specifically request public comments on this standard by [INSERT DATE 30 DAYS FROM DATE OF PUBLICATION IN THE FEDERAL REGISTER]. In addition, OMB wants to stress that the guidelines published today should be understood as a beginning of an evolutionary process that will include draft agency guidelines, public comment, final agency guidelines, development of experience with OMB and agency guidelines, and continued refinement of both OMB and agency guidelines. OMB modeled the draft definitions of Ainformation,@ Agovernment information,@ Ainformation dissemination product,@ and Adissemination@ on the longstanding definitions of those terms in OMB Circular A-130, but tailored them to fit into the context of these guidelines. Information that is disseminated on behalf of an agency (through a contract or a grant) is considered to be sponsored by the agency and is subject to these guidelines. Consistent with the PRA concept of agency Asponsorship@ of a collection of information, information is considered to be disseminated on behalf of an agency by a contractor or grantee if the dissemination is done at the agency=s specific request or with the agency=s specific approval. See 5 C.F.R. 1320.3(d). Finally, it should be noted that these guidelines focus primarily on the dissemination of substantive information (i.e. reports, studies, summaries) rather than information pertaining to basic agency operations. We have clarified two terms for the final guidelines. The proposed guidelines included Aopinions@ in the definition of Ainformation.@ We agree with comments that indicated agencies should not be accountable for correcting someone=s opinion, but in the agency=s presentation of the information, it should be clear that what is being offered is someone=s opinion rather than facts or the agency=s views. AOpinion@ has therefore been removed from the definition of Ainformation@ in the final guidelines. The definition for Adissemination@ was also revised after discussions with two Federal agencies that correspond frequently with individual members of the public regarding their participation in the agency=s programs. In addition, in the definition of Adissemination,@ we changed the exclusion for Ajudicial process@ to Aadjudicative process@ to make it clear that these guidelines do not apply to the issuance of agency adjudicative decisions. Reporting Requirements. Agencies have two reporting requirements. The first report, taking effect no later than one year after the issuance of these OMB guidelines, must provide the agency=s information quality guidelines that describe administrative mechanisms allowing affected persons to seek and obtain, where appropriate, correction of disseminated information that does not comply with these OMB guidelines. During the year that agencies have to complete their agency guidelines, agencies must publish the draft reports in the Federal Register for a period of public comment, and no later than nine months after the issuance of OMB=s guidelines, submit their draft reports to OMB for review. Upon completion of OMB=s review, final agency guidelines must be published in the Federal Register and made available through the agency website. The entire process must be completed by no later than one year after the issuance of the OMB guidance (no later than October 1, 2002). The second report is an annual fiscal year report to OMB (to be first submitted on January 1, 2004) providing information on the number, nature, and resolution of complaints received by the agency regarding its perceived or confirmed failure to comply with these OMB and agency guidelines. Regarding the proposed guidelines, we received detailed comments on the required report to OMB describing the number and nature of complaints received by the agency and how such complaints were resolved. Two Federal agencies stated that it would be burdensome to report to OMB on every single complaint they received and responded to, particularly because many of the complaints may be received in phone calls and given informal responses that address the callers= concerns. Recognizing that agencies may deal with large volumes of complaints on particular types of information disseminated by the agency, OMB=s guidelines allow the agency to provide qualitative and/or quantitative descriptions of complaints received and how they were resolved (or not). OMB also recognizes that a large number of comments about a specific document may only demonstrate that the information is controversial, not that its quality is flawed. In conclusion, issuance of these final guidelines meets the statutory requirement that section 515 imposed on OMB. As we stated earlier in this preamble, and in connection with the proposed guidelines, OMB has sought in developing these guidelines to make them flexible enough so that Federal agencies can apply them in a common sense, workable, and appropriately tailored manner to the wide variety of dissemination activities that the Federal Government undertakes. In addition, in drafting guidelines that will apply on a government-wide basis, OMB has been sensitive to the problem of unintended consequences and has tried to anticipate and address issues that could arise during the implementation of these guidelines. In this respect, the public and agency comments that we received on the proposed guidelines were very helpful and are greatly appreciated. As we explained above, we made a number of revisions to the guidelines to address the concerns raised in the comments, and we also believe that these and other concerns can be addressed as well in the implementing guidelines that each agency will develop in the coming months. In addition, OMB is issuing the Acapable of being substantially reproduced@ standard (paragraphs V.3.B, V.9, and V.10) on an interim final basis. We specifically request public comments on this standard over the next 30 days. Moreover, over time as the agencies and the public gain further experience with the OMB guidelines, we would appreciate receiving any suggestions for how OMB could improve them. Just as OMB requested public comment before issuing these final guidelines, OMB will refine these guidelines as experience develops and further public comment is obtained. Dated: September 24, 2001 ______________________________ John D. Graham Administrator, Office of Information and Regulatory Affairs Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies. I. OMB Responsibilities. Section 515 of the Treasury and General Government Appropriations Act for FY2001 (Public Law 106-554) directs the Office of Management and Budget to issue government-wide guidelines that provide policy and procedural guidance to Federal agencies for ensuring and maximizing the quality, objectivity, utility, and integrity of information, including statistical information, disseminated by Federal agencies. II. Agency Responsibilities. Section 515 directs agencies subject to the Paperwork Reduction Act (44 U.S.C. 3502(a)) to B 1. Issue their own information quality guidelines ensuring and maximizing the quality, objectivity, utility, and integrity of information, including statistical information, disseminated by the agency no later than one year after the date of issuance of the OMB guidelines; 2. Establish administrative mechanisms allowing affected persons to seek and obtain correction of information maintained and disseminated by the agency that does not comply with these OMB guidelines; and 3. Report to the Director of OMB the number and nature of complaints received by the agency regarding agency compliance with these OMB guidelines concerning the quality, objectivity, utility, and integrity of information and how such complaints were resolved. III. Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies. 1. Overall, agencies shall adopt a basic standard of quality (including objectivity, utility, and integrity) as a performance goal and should take appropriate steps to incorporate information quality criteria into agency information dissemination practices. Quality is to be ensured and established at levels appropriate to the nature and timeliness of the information to be disseminated. Agencies shall adopt specific standards of quality that are appropriate for the various categories of information they disseminate. 2. As a matter of good and effective agency information resources management, agencies shall develop a process for reviewing the quality (including the objectivity, utility, and integrity) of information before it is disseminated. Agencies shall treat information quality as integral to every step of an agency's development of information, including creation, collection, maintenance, and dissemination. This process shall enable the agency to substantiate the quality of the information it has disseminated through documentation or other means appropriate to the information. 3. To facilitate citizen review, agencies shall establish administrative mechanisms allowing affected persons to seek and obtain, where appropriate, timely correction of information maintained and disseminated by the agency that does not comply with OMB or agency guidelines. These administrative mechanisms shall be flexible, appropriate to the nature and timeliness of the disseminated information, and incorporated into agency information resources management and administrative practices. 4. The agency=s pre-dissemination review, under paragraph III.2, shall apply to information that the agency first disseminates on or after October 1, 2002. The agency=s administrative mechanisms, under paragraph III.3, shall apply to information that the agency disseminates on or after October 1, 2002, regardless of when the agency first disseminated the information. IV. Agency Reporting Requirements. 1. Agencies must designate the Chief Information Officer or another official to be responsible for agency compliance with these guidelines. 2. The agency shall respond to complaints in a manner appropriate to the nature and extent of the complaint. Examples of appropriate responses include personal contacts via letter or telephone, form letters, press releases or mass mailings that correct a widely disseminated error or address a frequently raised complaint. 3. Each agency must prepare a draft report, no later than April 1, 2002, providing the agency=s information quality guidelines and explaining how such guidelines will ensure and maximize the quality, objectivity, utility, and integrity of information, including statistical information, disseminated by the agency. This report must also detail the administrative mechanisms developed by that agency to allow affected persons to seek and obtain appropriate correction of information maintained and disseminated by the agency that does not comply with the OMB or the agency guidelines. 4. The agency must publish a notice of availability of this draft report in the Federal Register, and post this report on the agency=s website, to provide an opportunity for public comment. 5. Upon consideration of public comment and after appropriate revision, the agency must submit this draft report to OMB for review regarding consistency with these OMB guidelines no later than July 1, 2002. Upon completion of that OMB review and completion of this report, agencies must publish notice of the availability of this report in the Federal Register, and post this report on the agency=s web site no later than October 1, 2002. 6. On an annual fiscal-year basis, each agency must submit a report to the Director of OMB providing information (both quantitative and qualitative, where appropriate) on the number and nature of complaints received by the agency regarding agency compliance with these OMB guidelines and how such complaints were resolved. Agencies must submit these reports no later than January 1 of each following year, with the first report due January 1, 2004. V. Definitions. 1. AQuality@ is an encompassing term comprising utility, objectivity, and integrity. Therefore, the guidelines sometimes refer to these four statutory terms, collectively, as Aquality.@ 2. AUtility@ refers to the usefulness of the information to its intended users, including the public. In assessing the usefulness of information that the agency disseminates to the public, the agency needs to consider the uses of the information not only from the perspective of the agency but also from the perspective of the public. As a result, when reproducibility and transparency of information are relevant for assessing the information's usefulness from the public's perspective, the agency must take care to ensure that reproducibility and transparency have been addressed in its review of the information. 3. AObjectivity@ involves two distinct elements, presentation and substance. A. AObjectivity@ includes whether disseminated information is being presented in an accurate, clear, complete, and unbiased manner. This involves whether the information is presented within a proper context. Sometimes, in disseminating certain types of information to the public, other information must also be disseminated in order to ensure an accurate, clear, complete, and unbiased presentation. Also, the agency needs to identify the sources of the disseminated information (to the extent possible, consistent with confidentiality protections) and, in a scientific or statistical context, the supporting data and models, so that the public can assess for itself whether there may be some reason to question the objectivity of the sources. Where appropriate, supporting data should have full, accurate, transparent documentation, and error sources affecting data quality should be identified and disclosed to users. B. In addition, Aobjectivity@ involves a focus on ensuring accurate, reliable, and unbiased information. In a scientific or statistical context, the original or supporting data shall be generated, and the analytical results shall be developed, using sound statistical and research methods. i. If the results have been subject to formal, independent, external peer review, the information can generally be considered of acceptable objectivity. ii. In those situations involving influential scientific or statistical information, the results must be capable of being substantially reproduced, if the original or supporting data are independently analyzed using the same models. Reproducibility does not mean that the original or supporting data have to be capable of being replicated through new experiments, samples or tests. iii. Making the data and models publicly available will assist in determining whether analytical results are capable of being substantially reproduced. However, these guidelines do not alter the otherwise applicable standards and procedures for determining when and how information is disclosed. Thus, the objectivity standard does not override other compelling interests, such as privacy, trade secret, and other confidentiality protections. 4. AIntegrity@ refers to the security of information -- protection of the information from unauthorized access or revision, to ensure that the information is not compromised through corruption or falsification. 5. AInformation@ means any communication or representation of knowledge such as facts or data, in any medium or form, including textual, numerical, graphic, cartographic, narrative, or audiovisual forms. This definition includes information that an agency disseminates from a web page, but does not include the provision of hyperlinks to information that others disseminate. This definition does not include opinions, where the agency=s presentation makes it clear that what is being offered is someone=s opinion rather than fact or the agency=s views. 6. AGovernment information@ means information created, collected, processed, disseminated, or disposed of by or for the Federal Government. 7. Information dissemination product@ means any book, paper, map, machine-readable material, audiovisual production, or other documentary material, regardless of physical form or characteristic, an agency disseminates to the public. This definition includes any electronic document, CD-ROM, or web page. 8. Dissemination@ means agency initiated or sponsored distribution of information to the public (see 5 C.F.R. 1320.3(d) (definition of AConduct or Sponsor@). Dissemination does not include distribution limited to government employees or agency contractors or grantees; intra- or inter-agency use or sharing of government information; and responses to requests for agency records under the Freedom of Information Act, the Privacy Act, the Federal Advisory Committee Act or other similar law. This definition also does not include distribution limited to correspondence with individuals or persons, press releases, archival records, public filings, subpoenas or adjudicative processes. 9. Influential@ when used in the phrase influential scientific or statistical information@ means the agency expects that information in the form of analytical results will likely have an important effect on the development of domestic or international government or private sector policies or will likely have important consequences for specific technologies, substances, products or firms. 10. Capable of being substantially reproduced@ means that independent reanalysis of the original or supporting data using the same methods would generate similar analytical results, subject to an acceptable degree of imprecision.