[ecco-support] Improved: using wget to download multiple files on the ECCO drive

Thu Apr 9 12:15:03 EDT 2020

Martin et al.,

I made a short document showing how to use wget to download multiple files and/or directories from the ECCO drive.  

You can find the document here: 
https://drive.google.com/open?id=1XqQsfV1KnEX_ejQkcJoNrv6Qxknt8Kcr

We will also add a link to the document on the ECCO website.  

If you encounter any problems with the instructions or have any comments that you think others would find helpful we will incorporate them!

-Ian

On 4/9/20, 7:35 AM, "Fenty, Ian G (US 329C)" <ian.fenty at jpl.nasa.gov> wrote:

    Dear Martin,

    I will respond to your question (even though you have found a workaround by mounting the ECCO drive) because it might help others.

    One can use wget to download multiple files and directories from the ECCO drive.  Here are two examples:

    Ex 1) Download all of the lat-lon ETAN files from ECCOv4 Release 4:
    -------------------------------------------------------------------------------------------------
    wget --user=ifenty --password=ABCDEFGHIJKLMNOP -r -nc -np -nH https://ecco.jpl.nasa.gov/drive/files/Version4/Release4/interp_monthly/ETAN/

    The result will be a set of directories and files like this:
    	drive/files/Version4/Release4/interp_monthly/ETAN/1992/ETAN_1992_01.nc
    	drive/files/Version4/Release4/interp_monthly/ETAN/1992/ETAN_1992_02.nc
    	...
    	drive/files/Version4/Release4/interp_monthly/ETAN/2017/ETAN_2017_12.nc

    Ex 2) Download the entire ECCOv4 Release 4 solution:
    -------------------------------------------------------------------------------
    wget --user=ifenty --password=ABCDEFGHIJKLMNOP -r -nc -np -nH https://ecco.jpl.nasa.gov/drive/files/Version4/Release4/

    The result will be a set of directories and files like this:
    	drive/files/Version4/Release4/interp_monthly/ETAN/1992/ETAN_1992_01.nc
    	drive/files/Version4/Release4/interp_monthly/ETAN/1992/ETAN_1992_02.nc
    	...
    	drive/files/Version4/Release4/interp_monthly/ETAN/2017/ETAN_2017_11.nc

    Ex 3) Download all of the SALT files from the cube92 solution:
    ----------------------------------------------------------------------------------
    wget --user=ifenty --password=ABCDEFGHIJKLMNOP -r -nc -np -nH https://ecco.jpl.nasa.gov/drive/files/ECCO2/cube92_latlon_quart_90S90N/SALT.nc

    The result will be a set of directories and files like this:
    	drive/files/ECCO2/cube92_latlon_quart_90S90N/SALT.nc/SALT.1440x720x50.19920102.nc
    	drive/files/ECCO2/cube92_latlon_quart_90S90N/SALT.nc/SALT.1440x720x50.19920105.nc
    	...

    ** Remember *** the username and password required for wget (and for mounting the ECCO drive) are your EOSDIS "WebDAV/Programmatic API credentials".  These are DIFFERENT than your NASA EOSDIS username and password (yes, that is annoying).

    Find your WebDAV/Programmatic API credentials by navigating to: https://ecco.jpl.nasa.gov/drive/, then using your EOSDIS credentials to log on ("EARTHDATA LOGIN").  You will be taken to a page with your "ECCO Drive" login and password, these are your "WebDAV/Programmatic API credentials".  The page should look like the attached ECCO_DRIVE_CREDENTIALS_2.png image.  

    Alternatively, if you are taken to a page showing the ECCO Drive top-level directory then click on the big blue box called "Back to WebDAV Credentials" (see attached ECCO_DRIVE_CREDENTIALS.png).  You should then be taken to the "ECCO Drive" login and password page.

    For your reference, the "wget" options I used do the following:

    -r : Turn on recursive retrieving  (download files starting from a higher level)
    -np : "no parent" Do not ascend to parent directories when recursively downloading (download from subdirectories)
    -nc : "no clobber" yes to overwriting files in local directory if they are present 
    -nH: "no host directories" Do not generate "host-prefixed directories" (otherwise wget will make directories starting with the ECCO drive url).

    Another good option to consider is  
    --cut-dirs=2 do not create the first two subdirectories (/drive/files/) which are useless.  Instead start from /Version4/ or /ECCO2/ or whatever.

    I hope this is helpful!
    Ian

    On 4/8/20, 11:58 PM, "ecco-support-bounces at mit.edu on behalf of Martin Schmidt" <ecco-support-bounces at mit.edu on behalf of martin.schmidt at io-warnemuende.de> wrote:

        Dear ecco team,

        I am using cube92 as boundary data for a regional model. Thanks, for 
        making the data available. It helps a lot.

        I would like to continue an existing experiment until "now". My old 
        download procedure with wget from the ftp-server does not work any more.

        I have reactivated my NASA account and can visit again the ecco drive 
        for examples.

        https://ecco.jpl.nasa.gov/drive/files/ECCO2/cube92_latlon_quart_90S90N/SALT.nc

        I can also download single files from there. But this is not an option 
        for practical purpose.

        I am trying

        wget --user=mschmidt --ask-password 
        https://ecco.jpl.nasa.gov/drive/files/ECCO2/cube92_latlon_quart_90S90N/SALT.nc/SALT.1440x720.20200109.nc

        The response is

        Resolving ecco.jpl.nasa.gov (ecco.jpl.nasa.gov)... 128.149.52.112
        Connecting to ecco.jpl.nasa.gov 
        (ecco.jpl.nasa.gov)|128.149.52.112|:443... connected.
        HTTP request sent, awaiting response... 401 Unauthorized
        Authentication selected: Basic realm="ECCO_Drive"
        Reusing existing connection to ecco.jpl.nasa.gov:443.
        HTTP request sent, awaiting response... 401 Unauthorized

        The same with --http-user and --ftp-user.

        Do you have a suggestion, how to proceed?

        Kind regards,

        Martin

        _______________________________________________
        ecco-support mailing list
        ecco-support at mit.edu
        http://mailman.mit.edu/mailman/listinfo/ecco-support