[UW-GIS-L] NOAA Weather Data Help

[phil philhurvitz.com]

Emily,

You could use a R framework for doing a mostly automated method of handling the data that the NOAA people told you about. Documentation here: https://www1.ncdc.noaa.gov/pub/data/ghcn/daily/readme.txt indicates there is a list of stations, https://www.ncei.noaa.gov/data/daily-summaries/doc/ghcnd-stations.txt. The first one listed for Seattle is US1WAKG0006. It would be possible to parse that file and turn the lat/long values into points and then select the ones within WA. Then using that list of records, download the daily files for each station, then extract data from each file. You would still need to deal with linking these points with ZIP codes, which is not really a straightforward process for various reasons it is probably not worth going into at this time.

However, I think the NOAA folks may have steered you wrong … unless there is a different set of files that are known to have temperature values. Here is a worked up example in R

library(curl)
library(R.utils)
library(tools)

# temp dir
setwd(Sys.getenv("TEMP"))

f <- function(url){
    # get the file name from the URL
    myGzFname <- basename(url)
    # strip the GZ
    myFname <- file_path_sans_ext(myGzFname)

    # dowload and gunzip if necessary
    if(!file.exists(myFname)){
        # get a file
        curl_download(url = myUrl, destfile = myGzFname)

        # gunzip it
        gunzip(myGzFname)
    }

    # read it in
    x <- read.csv(myFname, header = FALSE)

    # look at it a bit
    message("first lines")
    print(head(x))

    # what is in V3?
    message("what is in V3?")
    print(table(x$V3))
}

The function downloads, unzips, and loads a file, then prints the first few lines and also prints a count of the climate variable records by measurement type. Run on a few files:




> f(ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/daily/by_station/US1WAKG0006.csv.gz)


first lines

           V1       V2   V3 V4   V5 V6 V7  V8

1 US1WAKG0006 20080605 PRCP  3   NA     N 700

2 US1WAKG0006 20080606 PRCP 38   NA     N 700

3 US1WAKG0006 20080607 PRCP 20   NA     N 530

4 US1WAKG0006 20080608 PRCP  0 TRUE     N 700

5 US1WAKG0006 20080609 PRCP  0   NA     N 650

6 US1WAKG0006 20080610 PRCP 33   NA     N 700

what is in V3?



DAPR MDPR PRCP SNOW SNWD WESD WESF

   7    7  283  274   87   93   75

There is no “TMAX” here, so maximum daily temperature seems to be missing. Same for this:



> f(ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/daily/by_station/US1WAKG0020.csv.gz)


first lines

           V1       V2   V3 V4 V5 V6 V7  V8

1 US1WAKG0020 20080619 PRCP  3 NA     N 700

2 US1WAKG0020 20080620 PRCP  0 NA     N 805

3 US1WAKG0020 20080622 PRCP  0 NA     N 810

4 US1WAKG0020 20080623 PRCP  0 NA     N 705

5 US1WAKG0020 20080624 PRCP  0 NA     N 654

6 US1WAKG0020 20080625 PRCP  0 NA     N 700

what is in V3?



DAPR MDPR PRCP SNOW

  58   58  391  250

So, unless there is some way of knowing which files contain max temperature values, this approach, while workable, doesn’t look like it will give you the data you are interested in.


________________________________
Phil Hurvitz phil at philhurvitz.com<mailto:phil at philhurvitz.com>
________________________________

From: Uw-gis-l <uw-gis-l-bounces at mailman13.u.washington.edu> On Behalf Of Luke Rogers
Sent: Thursday, November 2, 2023 12:17
To: uw-gis-l at u.washington.edu
Subject: Re: [UW-GIS-L] NOAA Weather Data Help

Thank you to Steven Walters for pointing out that this is the wrong PRISM! Here is the correct one:

https://www.prism.oregonstate.edu/

-Luke

From: Uw-gis-l <uw-gis-l-bounces at mailman13.u.washington.edu<mailto:uw-gis-l-bounces at mailman13.u.washington.edu>> On Behalf Of Luke Rogers
Sent: Thursday, November 2, 2023 11:05 AM
To: Emily Dalgo <edalgo at uw.edu<mailto:edalgo at uw.edu>>; uw-gis-l at u.washington.edu<mailto:uw-gis-l at u.washington.edu>
Subject: Re: [UW-GIS-L] NOAA Weather Data Help

You might try Climate NA or Prism.

https://climatena.ca/<https://urldefense.com/v3/__https:/climatena.ca/__;!!K-Hz7m0Vt54!iZJZ45Zc4yt6B5vrgVpg1_zJjHnYT_8uzSz6CNjMseiyOL0WkIA2qoPTCv9WQfHuJYoQqSJ6fJtNQbykIZMaUXMk$>
http://www.prism.washington.edu/home

-Luke

From: Uw-gis-l <uw-gis-l-bounces at mailman13.u.washington.edu<mailto:uw-gis-l-bounces at mailman13.u.washington.edu>> On Behalf Of Emily Dalgo
Sent: Thursday, November 2, 2023 11:00 AM
To: uw-gis-l at u.washington.edu<mailto:uw-gis-l at u.washington.edu>
Subject: [UW-GIS-L] NOAA Weather Data Help

Good morning all,

I am working on a MA thesis related to the association between weather patterns and mortality. Ideally, I would like to have daily high temperatures for all zip codes in the state of Washington for all days between 2016 and 2021 (years for which I have mortality data).

I have been doing data entry by hand using the search tool at https://www.ncdc.noaa.gov/cdo-web/search<https://urldefense.com/v3/__https:/www.ncdc.noaa.gov/cdo-web/search__;!!K-Hz7m0Vt54!ims3pyNFjW9flVXPC5-KDskBpgeT-bcJoCjA1UixOFSRLlsVyQSunD-YbENcQwUMjNoHqmtLc7fstOA$>. The tool does not give weather data for zip codes containing no weather station, so I have been manually hand coding the nearest weather station using the the tools at https://www.ncei.noaa.gov/maps/daily-summaries/<https://urldefense.com/v3/__https:/www.ncei.noaa.gov/maps/daily-summaries/__;!!K-Hz7m0Vt54!ims3pyNFjW9flVXPC5-KDskBpgeT-bcJoCjA1UixOFSRLlsVyQSunD-YbENcQwUMjNoHqmtLTICGMEI$>

I spoke to people at NOAA who said the data I need might be in the files at https://www.ncei.noaa.gov/data/daily-summaries/access/<https://urldefense.com/v3/__https:/www.ncei.noaa.gov/data/daily-summaries/access/__;!!K-Hz7m0Vt54!ims3pyNFjW9flVXPC5-KDskBpgeT-bcJoCjA1UixOFSRLlsVyQSunD-YbENcQwUMjNoHqmtLigOyH3c$> but I have had no luck accessing the files because of their massive size.

Do you have any tips for working with the NOAA metadata? If not, do you know of a different easily-accessible source of data on high temperatures by day (or even by weather station that can be linked to zip codes)?

Thanks in advance for any ideas about this.

Emily Dalgo


--
Emily Dalgo
Graduate Student, Department of Sociology
University of Washington
edalgo at uw.edu<mailto:edalgo at uw.edu>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/uw-gis-l/attachments/20231105/0ca6875d/attachment.html>