Azimuth Backup Project (Part 2)

I want to list some databases that are particularly worth backing up. But to do this, we need to know what’s already been backed up. That’s what this post is about.

Azimuth backups

Here is information as of now (21:45 GMT 20 December 2016). I won’t update this information. For up-to-date information see

Azimuth Backup Project: Issue Tracker.

For up-to-date information on the progress of each of individual databases listed below, click on my summary of what’s happening now.

Here are the databases that we’ve backed up:

• NASA GISTEMP website at http://data.giss.nasa.gov/gistemp/downloaded by Jan and uploaded to Sakari’s datarefuge server.

• NOAA Carbon Dioxide Information Analysis Center (CDIAC) data at ftp.ncdc.noaa.gov/pub/data/paleo/cdiac.ornl.gov-pub — downloaded by Jan and uploaded to Sakari’s datarefuge server.

• NOAA Carbon Tracker website at http://www.esrl.noaa.gov/psd/data/gridded/data.carbontracker.htmldownloaded by Jan, uploaded to Sakari’s datarefuge server.

These are still in progress, but I think we have our hands on the data:

• NOAA Precipitation Frequency Data at http://hdsc.nws.noaa.gov/hdsc/pfds/ and ftp://hdsc.nws.noaa.gov/pubdownloaded by Borislav, not yet uploaded to Sakari’s datarefuge server.

• NOAA Carbon Dioxide Information Analysis Center (CDIAC) website at http://cdiac.ornl.govdownloaded by Jan, uploaded to Sakari’s datarefuge server, but there’s evidence that the process was incomplete.

• NOAA website at https://www.ncdc.noaa.govdownloaded by Jan, who is now attempting to upload it to Sakari’s datarefuge server.

• NOAA National Centers for Environmental Information (NCEI) website at https://www.ncdc.noaa.govdownloaded by Jan, who is now attempting to upload it to Sakari’s datarefuge server, but there are problems.

• Ocean and Atmospheric Research data at ftp.oar.noaa.gov — downloaded by Jan, now attempting to upload it to Sakari’s datarefuge server.

• NOAA NCEP/NCAR Reanalysis ftp site at ftp.cdc.noaa.gov/Datasets/ncep.reanalysis/ — downloaded by Jan, now attempting to upload it to Sakari’s datarefuge server.

I think we’re getting these now, more or less:

• NOAA National Centers for Environmental Information (NCEI) ftp site at ftp://eclipse.ncdc.noaa.gov/pub/ — in the process of being downloaded by Jan, “Very large. May be challenging to manage with my facilities”.

• NASA Planetary Data System (PDS) data at https://pds.nasa.govin the process of being downloaded by Sakari.

• NOAA tides and currents products website at https://tidesandcurrents.noaa.gov/products.html, which includes the sea level trends data at https://tidesandcurrents.noaa.gov/sltrends/sltrends.htmlJan is downloading this.

• NOAA National Centers for Environmental Information (NCEI) satellite datasets website at https://www.ncdc.noaa.gov/data-access/satellite-data/satellite-data-access-datasetsJan is downloading this.

• NASA JASON3 sea level data at http://sealevel.jpl.nasa.gov/missions/jason3/Jan is downloading this.

• U.S. Forest Service Climate Change Atlas website at http://www.fs.fed.us/nrs/atlas/Jan is downloading this.

• NOAA Global Monitoring Division website at http://www.esrl.noaa.gov/gmd/dv/ftpdata.htmlJan is downloading this.

• NOAA Global Monitoring Division ftp data at aftp.cmdl.noaa.gov/ — Jan is downloading this.

• NOAA National Data Buoy Center website at http://www.ndbc.noaa.gov/Jan is downloading this.

• NASA-ESDIS Oak Ridge National Laboratory Distributed Active Archive (DAAC) on Biogeochemical Dynamics at https://daac.ornl.gov/get_data.shtmlJan is downloading this.

• NASA-ESDIS Oak Ridge National Laboratory Distributed Active Archive (DAAC) on Biogeochemical Dynamics website at https://daac.ornl.gov/Jan is downloading this.

Other backups

Other backups are listed at

The Climate Mirror Project, https://climate.daknob.net/.

This nicely provides the sizes of various backups, and other useful information. Some are ‘signed and verified’ with cryptographic keys, but I’m not sure exactly what that means, and the details matter.

About 90 databases are listed here, along with some size information and some information about whether people have already backed them up or are in process:

Gov. Climate Datasets (Archive). (Click on the tiny word “Datasets” at the bottom of the page!)


azimuth_logo

7 Responses to Azimuth Backup Project (Part 2)

  1. John Baez says:

    Here are some US government databases that don’t seem to have been backed up yet by anyone. The National Snow and Ice Data Center is recording the melting of the Arctic as the Earth warms:

    • National Snow and Ice Data Center (NSIDC), data at https://nsidc.org/data

    This contains lots of data about clouds, atmospheric aerosols, the Earth’s radiation budget and tropospheric chemistry:

    • NASA Atmospheric Science Data Center (ASDC), ASDC products supported, at https://eosweb.larc.nasa.gov/projects-supported-list

    • John Baez says:

      Good news! I think it’s not urgent to back up the National Snow and Ice Data Center (NSIDC). Wikipedia says it’s one of 12 “Distributed Active Archive Centers” funded by the National Aeronautics and Space Administration to archive and distribute data from NASA’s past and current satellites and field measurement programs. It’s based at the University of Colorado Boulder Cooperative Institute for Research in Environmental Sciences (CIRES). So, I believe it is already serving as a kind of backup. Trump could shut down this program but I don’t think he could rapidly make the university delete this data. I’ll check.

      • nad says:

        Trump could shut down this program but I don’t think he could rapidly make the university delete this data.

        Who knows it’s a public university isn’t it?
        Anyways I can only repeat what you do is no “backup” but rather an “emergency measure” and I repeat that I think this should be more communicated as such.
        In particular as you know disrupting measurements may be one major problem. The Verge had an article about following an op-ed written by Robert S. Walker, former chairman of the House Science Committee, and Peter Navarro, an economist and public policy expert. were they state:

        Whether or not Trump will completely slash NASA’s Earth Science division is unknown, but it’s not impossible.

        For example the SORCE mission, I just mentioned on the forum when looking for important measurements is part of NASA’s Earth Science Program.

        I am still looking for similar measurements from other space agencies, but haven’t sofar found any. As I wrote I couldn’t find anything similar on russian pages and likewise I haven’t yet found anything on the ESA side.
        I dont know about about the chinese, but their
        primary satellite control facility in Xian isn’t even linked to the Wikipedia site.

      • John Baez says:

        Nad wrote:

        Who knows it’s a public university isn’t it?

        The United States is a federation of states, and the federal government has limited ability to control what individual states do. The University of Colorado is run by the state of Colorado, not the federal government. So, the only legal way Trump could force researchers there to delete data from their computers is to get Congress to pass a law saying they need to do this (or, for example, that they need to do this to continue to get funds from the federal government). This would be a much slower process than, say, Trump telling NASA to delete all climate databases from their servers.

        Of course, all this assumes that Trump doesn’t become the sort of dictator who destroys the existing rules. But that, too, would take some time to happen. Our goal is to get some things done by January 20th, to prevent certain kinds of rapid destruction of data.

        Anyways I can only repeat what you do is no “backup” but rather an “emergency measure” and I repeat that I think this should be more communicated as such.

        It’s obviously an emergency measure; nobody in the US needs to be told that. Every day brings scary news about the political situation in this country.

  2. Dot Field says:

    @JohnBaez Great job! Don’t forget to also backup things at the Internet Archive (https://archive.org).

    Also everyone is centered around backing up climate data, but we should also backup Trump’s own nonsense on climate change (as evidence to add up to his countless lawsuits… who knows? Even Obama is being sued right now for climate change related topics.), so here they are:

    https://web.archive.org/web/https:/twitter.com/realdonaldtrump/status/265895292191248385
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/418542137899491328
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/349973299889057792
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/316252016190054400
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/475668993928212480
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/435574043354611712
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/270628609817976834
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/435393088383889408
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/412159674042294272
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/326875628966117376
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/349973845228269569
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/512246203967619072
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/338448296022511618
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/488825209189711873
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/427226424987385856
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/417818392826232832
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/488926006225285120
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/431018674695442432
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/428418323660165120
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/653385381526806528
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/404420095113715712
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/408977616926830592
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/319377285687939072
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/428416406280241153
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/408380302206443520
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/521862351218573312
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/489381851350319107
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/407505938774757376
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/568387798924963840
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/493935815207043072
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/420333882597466112
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/450964791985971200
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/326874524576526337
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/422819593120256000
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/568021533131718656
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/408018451362766849
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/416909004984844288
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/334254335116587008
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/535102735830773760
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/338978381636984832
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/428954382915223552
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/417816035107299328
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/264010129106665472
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/488813607958757376
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/264007296970018816
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/427556692109574146
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/412162068989874176
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/372781203239104512
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/440811151283486720
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/326781792340299776
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/408983789830815744
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/416539702096052224
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/338429342646423553
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/402217536751951872
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/423179182198104064
    https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/314744479821205505

  3. Why was this not done years earlier?

You can use Markdown or HTML in your comments. You can also use LaTeX, like this: $latex E = m c^2 $. The word 'latex' comes right after the first dollar sign, with a space after it.

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s