[ecco-support] [EXTERNAL] wget access to cub92 data

Thu Apr 9 10:35:43 EDT 2020

Dear Martin,

I will respond to your question (even though you have found a workaround by mounting the ECCO drive) because it might help others.

One can use wget to download multiple files and directories from the ECCO drive.  Here are two examples:

Ex 1) Download all of the lat-lon ETAN files from ECCOv4 Release 4:
-------------------------------------------------------------------------------------------------
wget --user=ifenty --password=ABCDEFGHIJKLMNOP -r -nc -np -nH https://ecco.jpl.nasa.gov/drive/files/Version4/Release4/interp_monthly/ETAN/

The result will be a set of directories and files like this:
	drive/files/Version4/Release4/interp_monthly/ETAN/1992/ETAN_1992_01.nc
	drive/files/Version4/Release4/interp_monthly/ETAN/1992/ETAN_1992_02.nc
	...
	drive/files/Version4/Release4/interp_monthly/ETAN/2017/ETAN_2017_12.nc

Ex 2) Download the entire ECCOv4 Release 4 solution:
-------------------------------------------------------------------------------
wget --user=ifenty --password=ABCDEFGHIJKLMNOP -r -nc -np -nH https://ecco.jpl.nasa.gov/drive/files/Version4/Release4/

The result will be a set of directories and files like this:
	drive/files/Version4/Release4/interp_monthly/ETAN/1992/ETAN_1992_01.nc
	drive/files/Version4/Release4/interp_monthly/ETAN/1992/ETAN_1992_02.nc
	...
	drive/files/Version4/Release4/interp_monthly/ETAN/2017/ETAN_2017_11.nc

Ex 3) Download all of the SALT files from the cube92 solution:
----------------------------------------------------------------------------------
wget --user=ifenty --password=ABCDEFGHIJKLMNOP -r -nc -np -nH https://ecco.jpl.nasa.gov/drive/files/ECCO2/cube92_latlon_quart_90S90N/SALT.nc

The result will be a set of directories and files like this:
	drive/files/ECCO2/cube92_latlon_quart_90S90N/SALT.nc/SALT.1440x720x50.19920102.nc
	drive/files/ECCO2/cube92_latlon_quart_90S90N/SALT.nc/SALT.1440x720x50.19920105.nc
	...

** Remember *** the username and password required for wget (and for mounting the ECCO drive) are your EOSDIS "WebDAV/Programmatic API credentials".  These are DIFFERENT than your NASA EOSDIS username and password (yes, that is annoying).

Find your WebDAV/Programmatic API credentials by navigating to: https://ecco.jpl.nasa.gov/drive/, then using your EOSDIS credentials to log on ("EARTHDATA LOGIN").  You will be taken to a page with your "ECCO Drive" login and password, these are your "WebDAV/Programmatic API credentials".  The page should look like the attached ECCO_DRIVE_CREDENTIALS_2.png image.  

Alternatively, if you are taken to a page showing the ECCO Drive top-level directory then click on the big blue box called "Back to WebDAV Credentials" (see attached ECCO_DRIVE_CREDENTIALS.png).  You should then be taken to the "ECCO Drive" login and password page.

For your reference, the "wget" options I used do the following:

-r : Turn on recursive retrieving  (download files starting from a higher level)
-np : "no parent" Do not ascend to parent directories when recursively downloading (download from subdirectories)
-nc : "no clobber" yes to overwriting files in local directory if they are present 
-nH: "no host directories" Do not generate "host-prefixed directories" (otherwise wget will make directories starting with the ECCO drive url).

Another good option to consider is  
--cut-dirs=2 do not create the first two subdirectories (/drive/files/) which are useless.  Instead start from /Version4/ or /ECCO2/ or whatever.

I hope this is helpful!
Ian

On 4/8/20, 11:58 PM, "ecco-support-bounces at mit.edu on behalf of Martin Schmidt" <ecco-support-bounces at mit.edu on behalf of martin.schmidt at io-warnemuende.de> wrote:

    Dear ecco team,

    I am using cube92 as boundary data for a regional model. Thanks, for 
    making the data available. It helps a lot.

    I would like to continue an existing experiment until "now". My old 
    download procedure with wget from the ftp-server does not work any more.

    I have reactivated my NASA account and can visit again the ecco drive 
    for examples.

    https://ecco.jpl.nasa.gov/drive/files/ECCO2/cube92_latlon_quart_90S90N/SALT.nc

    I can also download single files from there. But this is not an option 
    for practical purpose.

    I am trying

    wget --user=mschmidt --ask-password 
    https://ecco.jpl.nasa.gov/drive/files/ECCO2/cube92_latlon_quart_90S90N/SALT.nc/SALT.1440x720.20200109.nc

    The response is

    Resolving ecco.jpl.nasa.gov (ecco.jpl.nasa.gov)... 128.149.52.112
    Connecting to ecco.jpl.nasa.gov 
    (ecco.jpl.nasa.gov)|128.149.52.112|:443... connected.
    HTTP request sent, awaiting response... 401 Unauthorized
    Authentication selected: Basic realm="ECCO_Drive"
    Reusing existing connection to ecco.jpl.nasa.gov:443.
    HTTP request sent, awaiting response... 401 Unauthorized

    The same with --http-user and --ftp-user.

    Do you have a suggestion, how to proceed?

    Kind regards,

    Martin

    _______________________________________________
    ecco-support mailing list
    ecco-support at mit.edu
    http://mailman.mit.edu/mailman/listinfo/ecco-support

-------------- next part --------------
A non-text attachment was scrubbed...
Name: ECCO_DRIVE_CREDENTIALS.png
Type: image/png
Size: 288594 bytes
Desc: ECCO_DRIVE_CREDENTIALS.png
Url : http://mailman.mit.edu/pipermail/ecco-support/attachments/20200409/8142e3b5/attachment-0002.png
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ECCO_DRIVE_CREDENTIALS_2.png
Type: image/png
Size: 278768 bytes
Desc: ECCO_DRIVE_CREDENTIALS_2.png
Url : http://mailman.mit.edu/pipermail/ecco-support/attachments/20200409/8142e3b5/attachment-0003.png