記錄一下 下載EBAS-Thredds數(shù)據(jù)的流程, 以全站點(diǎn)的ozone為例

1. 下載ebas-thredds-harvest的腳本

https://git.nilu.no/ebas/ebas-thredds-harvest

2. 獲取所有的文件名列表 => EBAS.filelist.txt

從這里復(fù)制過(guò)來(lái), 并掐頭去尾, 保持每一行格式固定,都是文件名

3. 獲取含ozone的文件名列表 => EBAS.filelist.ozone.txt

# in wsl
grep ozone EBAS.filelist.txt > EBAS.filelist.ozone.txt

4. 獲取站點(diǎn)列表 => EBAS.sites.txt

# in wsl
awk -F. '{print $1}' EBAS.filelist.ozone.txt | sort | uniq > EBAS.sites.txt

5. python配置必要環(huán)境

netCDF4包, thredds包等等, 按需來(lái),

6. 執(zhí)行腳本獲取所有站點(diǎn)對(duì)應(yīng)的urls => urls.txt

執(zhí)行下面的powershell腳本, 循環(huán)調(diào)用python腳本, 獲取http下載鏈接, 并保存至urls.txt

這一步比較慢

# get_ozone_urls.ps1

$sites = "AM0001R", "AR0001R", "AR0002G", "AR0004R", "AR0005R", "AT0002R", "AT0003R", "AT0004R", "AT0005R", "AT0030R", "AT0032R", "AT0033R", "AT0034G", "AT0037R", "AT0038R", "AT0040R", "AT0041R", "AT0042R", "AT0043R", "AT0044R", "AT0045R", "AT0046R", "AT0047R", "AT0048R", "AT0049R", "AT0050R", "AU0002G", "BB0001R", "BE0001R", "BE0032R", "BE0035R", "BG0053R", "BM0001R", "BO0001R", "CA0002R", "CA0003R", "CA0006R", "CA0010R", "CA0011R", "CA0012R", "CA0023R", "CA0103R", "CA0420G", "CH0001G", "CH0002R", "CH0003R", "CH0004R", "CH0005R", "CH0010U", "CH0031R", "CH0053R", "CL0001R", "CV0001G", "CY0002R", "CZ0001R", "CZ0003R", "CZ0005R", "DE0001R", "DE0002R", "DE0003R", "DE0004R", "DE0005R", "DE0006R", "DE0007R", "DE0008R", "DE0009R", "DE0011R", "DE0012R", "DE0013R", "DE0014R", "DE0017R", "DE0018R", "DE0026R", "DE0031R", "DE0035R", "DE0038R", "DE0039R", "DE0042R", "DE0043G", "DE0045R", "DE0046R", "DE0047R", "DE0054R", "DE0057G", "DE0060G", "DK0005R", "DK0009R", "DK0010G", "DK0012R", "DK0025G", "DK0031R", "DK0032R", "DK0041R", "EE0009R", "EE0011R", "EG0001U", "EG0002U", "EG0004R", "ES0001R", "ES0002R", "ES0003R", "ES0004R", "ES0005R", "ES0006R", "ES0007R", "ES0008R", "ES0009R", "ES0010R", "ES0011R", "ES0012R", "ES0013R", "ES0014R", "ES0015R", "ES0016R", "ES0017R", "ES0018G", "FI0004R", "FI0009R", "FI0017R", "FI0018R", "FI0022R", "FI0037R", "FI0096G", "FR0008R", "FR0009R", "FR0010R", "FR0011R", "FR0012R", "FR0013R", "FR0014R", "FR0015R", "FR0016R", "FR0017R", "FR0018R", "FR0019R", "FR0020R", "FR0023R", "FR0024R", "FR0025R", "FR0026R", "FR0028R", "FR0030R", "FR0031R", "FR0032R", "FR0034G", "GB0002R", "GB0006R", "GB0013R", "GB0014R", "GB0015R", "GB0031R", "GB0033R", "GB0035R", "GB0036R", "GB0037R", "GB0038R", "GB0039R", "GB0041R", "GB0043R", "GB0044R", "GB0045R", "GB0048R", "GB0049R", "GB0050R", "GB0051R", "GB0052R", "GB0053R", "GB0059G", "GB1055R", "GR0001R", "GR0002R", "GR0003R", "GR0101R", "HU0002R", "HU0003R", "HU0017R", "ID1013R", "IE0001R", "IE0031R", "IS0091R", "IT0001R", "IT0004R", "IT0009R", "IT0014R", "IT0015U", "IT0016R", "IT0018R", "IT0019R", "JP0002G", "JP0003U", "JP1020R", "JP1028G", "JP1029R", "KE0001G", "KG0002U", "KR0100R", "KR0101R", "LT0015R", "LV0010R", "LV0016R", "LV2000U", "MK0007R", "MT0001R", "NL0007R", "NL0009R", "NL0010R", "NL0011R", "NL0091R", "NL0644R", "NO0001R", "NO0002R", "NO0015R", "NO0030R", "NO0039R", "NO0041R", "NO0042G", "NO0043R", "NO0044R", "NO0045R", "NO0047R", "NO0048R", "NO0052R", "NO0055R", "NO0056R", "NO0058G", "NO0059G", "NO0488R", "NO0489R", "NO0492R", "NO0762R", "NO0977R", "NO1007R", "NO1011R", "NO1083R", "NO1200R", "NP0001G", "NZ0001R", "NZ0002R", "NZ0003G", "PE0001R", "PL0002R", "PL0003R", "PL0004R", "PL0005R", "PT0004R", "RO0003R", "RO0008R", "RS0005R", "RU0001R", "RU0013R", "RU0016R", "RU0018R", "RU0100R", "SE0002R", "SE0003R", "SE0005R", "SE0011R", "SE0012R", "SE0013R", "SE0014R", "SE0018R", "SE0019R", "SE0020R", "SE0022R", "SE0032R", "SE0033R", "SE0034R", "SE0035R", "SE0039R", "SE0094R", "SI0008R", "SI0031R", "SI0032R", "SI0033R", "SK0002R", "SK0004R", "SK0005R", "SK0006R", "SK0007R", "US0008R", "US0039G", "US0142R", "US0204R", "US0602R", "US0901R", "US1200R", "US6001R", "US6004G", "US6005G", "US6006R", "VN0001R", "ZA0001G"

foreach ($s in $sites){
    echo $s
    python ebas-thredds-harvest/get_tds_data.py -s "ozone" -c "$s" -p "http" >> urls.txt
}

7. 下載這些鏈接

wget, curl之類的去下載urls.txt里的http鏈接即可

后續(xù)的處理邏輯記在EBAS觀測(cè)數(shù)據(jù)處理