I want to list some databases that are particularly worth backing up. But to do this, we need to know what’s already been backed up. That’s what this post is about.
Azimuth backups
Here is information as of now (21:45 GMT 20 December 2016). I won’t update this information. For up-to-date information see
• Azimuth Backup Project: Issue Tracker.
For up-to-date information on the progress of each of individual databases listed below, click on my summary of what’s happening now.
Here are the databases that we’ve backed up:
• NASA GISTEMP website at http://data.giss.nasa.gov/gistemp/ — downloaded by Jan and uploaded to Sakari’s datarefuge server.
• NOAA Carbon Dioxide Information Analysis Center (CDIAC) data at ftp.ncdc.noaa.gov/pub/data/paleo/cdiac.ornl.gov-pub — downloaded by Jan and uploaded to Sakari’s datarefuge server.
• NOAA Carbon Tracker website at http://www.esrl.noaa.gov/psd/data/gridded/data.carbontracker.html — downloaded by Jan, uploaded to Sakari’s datarefuge server.
These are still in progress, but I think we have our hands on the data:
• NOAA Precipitation Frequency Data at http://hdsc.nws.noaa.gov/hdsc/pfds/ and ftp://hdsc.nws.noaa.gov/pub — downloaded by Borislav, not yet uploaded to Sakari’s datarefuge server.
• NOAA Carbon Dioxide Information Analysis Center (CDIAC) website at http://cdiac.ornl.gov — downloaded by Jan, uploaded to Sakari’s datarefuge server, but there’s evidence that the process was incomplete.
• NOAA website at https://www.ncdc.noaa.gov — downloaded by Jan, who is now attempting to upload it to Sakari’s datarefuge server.
• NOAA National Centers for Environmental Information (NCEI) website at https://www.ncdc.noaa.gov — downloaded by Jan, who is now attempting to upload it to Sakari’s datarefuge server, but there are problems.
• Ocean and Atmospheric Research data at ftp.oar.noaa.gov — downloaded by Jan, now attempting to upload it to Sakari’s datarefuge server.
• NOAA NCEP/NCAR Reanalysis ftp site at ftp.cdc.noaa.gov/Datasets/ncep.reanalysis/ — downloaded by Jan, now attempting to upload it to Sakari’s datarefuge server.
I think we’re getting these now, more or less:
• NOAA National Centers for Environmental Information (NCEI) ftp site at ftp://eclipse.ncdc.noaa.gov/pub/ — in the process of being downloaded by Jan, “Very large. May be challenging to manage with my facilities”.
• NASA Planetary Data System (PDS) data at https://pds.nasa.gov — in the process of being downloaded by Sakari.
• NOAA tides and currents products website at https://tidesandcurrents.noaa.gov/products.html, which includes the sea level trends data at https://tidesandcurrents.noaa.gov/sltrends/sltrends.html — Jan is downloading this.
• NOAA National Centers for Environmental Information (NCEI) satellite datasets website at https://www.ncdc.noaa.gov/data-access/satellite-data/satellite-data-access-datasets — Jan is downloading this.
• NASA JASON3 sea level data at http://sealevel.jpl.nasa.gov/missions/jason3/ — Jan is downloading this.
• U.S. Forest Service Climate Change Atlas website at http://www.fs.fed.us/nrs/atlas/ — Jan is downloading this.
• NOAA Global Monitoring Division website at http://www.esrl.noaa.gov/gmd/dv/ftpdata.html — Jan is downloading this.
• NOAA Global Monitoring Division ftp data at aftp.cmdl.noaa.gov/ — Jan is downloading this.
• NOAA National Data Buoy Center website at http://www.ndbc.noaa.gov/ — Jan is downloading this.
• NASA-ESDIS Oak Ridge National Laboratory Distributed Active Archive (DAAC) on Biogeochemical Dynamics at https://daac.ornl.gov/get_data.shtml — Jan is downloading this.
• NASA-ESDIS Oak Ridge National Laboratory Distributed Active Archive (DAAC) on Biogeochemical Dynamics website at https://daac.ornl.gov/ — Jan is downloading this.
Other backups
Other backups are listed at
• The Climate Mirror Project, https://climate.daknob.net/.
This nicely provides the sizes of various backups, and other useful information. Some are ‘signed and verified’ with cryptographic keys, but I’m not sure exactly what that means, and the details matter.
About 90 databases are listed here, along with some size information and some information about whether people have already backed them up or are in process:
• Gov. Climate Datasets (Archive). (Click on the tiny word “Datasets” at the bottom of the page!)
Here are some US government databases that don’t seem to have been backed up yet by anyone. The National Snow and Ice Data Center is recording the melting of the Arctic as the Earth warms:
• National Snow and Ice Data Center (NSIDC), data at https://nsidc.org/data
This contains lots of data about clouds, atmospheric aerosols, the Earth’s radiation budget and tropospheric chemistry:
• NASA Atmospheric Science Data Center (ASDC), ASDC products supported, at https://eosweb.larc.nasa.gov/projects-supported-list
Good news! I think it’s not urgent to back up the National Snow and Ice Data Center (NSIDC). Wikipedia says it’s one of 12 “Distributed Active Archive Centers” funded by the National Aeronautics and Space Administration to archive and distribute data from NASA’s past and current satellites and field measurement programs. It’s based at the University of Colorado Boulder Cooperative Institute for Research in Environmental Sciences (CIRES). So, I believe it is already serving as a kind of backup. Trump could shut down this program but I don’t think he could rapidly make the university delete this data. I’ll check.
Who knows it’s a public university isn’t it?
Anyways I can only repeat what you do is no “backup” but rather an “emergency measure” and I repeat that I think this should be more communicated as such.
In particular as you know disrupting measurements may be one major problem. The Verge had an article about following an op-ed written by Robert S. Walker, former chairman of the House Science Committee, and Peter Navarro, an economist and public policy expert. were they state:
For example the SORCE mission, I just mentioned on the forum when looking for important measurements is part of NASA’s Earth Science Program.
I am still looking for similar measurements from other space agencies, but haven’t sofar found any. As I wrote I couldn’t find anything similar on russian pages and likewise I haven’t yet found anything on the ESA side.
I dont know about about the chinese, but their
primary satellite control facility in Xian isn’t even linked to the Wikipedia site.
Nad wrote:
The United States is a federation of states, and the federal government has limited ability to control what individual states do. The University of Colorado is run by the state of Colorado, not the federal government. So, the only legal way Trump could force researchers there to delete data from their computers is to get Congress to pass a law saying they need to do this (or, for example, that they need to do this to continue to get funds from the federal government). This would be a much slower process than, say, Trump telling NASA to delete all climate databases from their servers.
Of course, all this assumes that Trump doesn’t become the sort of dictator who destroys the existing rules. But that, too, would take some time to happen. Our goal is to get some things done by January 20th, to prevent certain kinds of rapid destruction of data.
It’s obviously an emergency measure; nobody in the US needs to be told that. Every day brings scary news about the political situation in this country.
@JohnBaez Great job! Don’t forget to also backup things at the Internet Archive (https://archive.org).
Also everyone is centered around backing up climate data, but we should also backup Trump’s own nonsense on climate change (as evidence to add up to his countless lawsuits… who knows? Even Obama is being sued right now for climate change related topics.), so here they are:
https://web.archive.org/web/https:/twitter.com/realdonaldtrump/status/265895292191248385
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/418542137899491328
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/349973299889057792
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/316252016190054400
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/475668993928212480
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/435574043354611712
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/270628609817976834
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/435393088383889408
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/412159674042294272
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/326875628966117376
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/349973845228269569
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/512246203967619072
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/338448296022511618
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/488825209189711873
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/427226424987385856
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/417818392826232832
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/488926006225285120
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/431018674695442432
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/428418323660165120
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/653385381526806528
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/404420095113715712
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/408977616926830592
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/319377285687939072
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/428416406280241153
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/408380302206443520
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/521862351218573312
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/489381851350319107
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/407505938774757376
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/568387798924963840
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/493935815207043072
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/420333882597466112
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/450964791985971200
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/326874524576526337
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/422819593120256000
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/568021533131718656
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/408018451362766849
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/416909004984844288
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/334254335116587008
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/535102735830773760
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/338978381636984832
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/428954382915223552
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/417816035107299328
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/264010129106665472
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/488813607958757376
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/264007296970018816
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/427556692109574146
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/412162068989874176
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/372781203239104512
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/440811151283486720
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/326781792340299776
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/408983789830815744
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/416539702096052224
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/338429342646423553
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/402217536751951872
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/423179182198104064
https://web.archive.org/web/https://twitter.com/realdonaldtrump/status/314744479821205505
In my post Saving climate data (part 2) I explained how people can nominate databases to be backed up by the Internet Archive. Here’s what I said:
Why was this not done years earlier?
Note to all: Something to help with.