[acs-r] Error message in acs 2.0

Tue Apr 19 09:29:20 EDT 2016

Just to chime in as well, both of these comments are right on target.  

The census API has a limit on the amount of data that can be requested
in a single call -- somewhere around 50 variables, I think.  When
given geo.sets with more than this many geographies (say, 591 tracts),
the acs.fetch function calls itself recursively to get circle through
and fetch it all, and then it simply combines them for you.

This generally works fine, but when the geo.sets get really large
(Carl's 550 tracts fits with my experience as well), the package runs
into some of R's internal limits on how deeply nested things can get
in calls like this.  In addition, even if you don't run into the
nesting issue, acs.fetch really lags when you get up into the hundreds
of geographies -- especially with the larger tables -- and I suspect
that sometimes this may cause timeouts on the API side.

In theory this can probably be fixed by changing options in the R
session to allow for more nesting (or smarter coding on the part of
the package developer -- sorry!), but in practice it's probably best
to do as Carl and Michael suggest: either break the call down into a
few subsets and then combine after fetching, or (better yet) use the
wildcard tracts="*", download all the tracts in the county in a single
(reasonably fast) fetch, and then filter (using the "geography()"
function on the fetched acs object).

The only other small point I'd make is that Carl's (otherwise)
excellent advice gives the wrong function to create the geo.set.
Where he has

> geo.set(state="IL", county="Cook", tract="*")

it should be 

> mygeo <- geo.make(state="IL",county="Cook" ,tract="*")

This can then be followed with:

> race <- acs.fetch(endyear=2014, span=5, geography = mygeo, table.number="B03002", dataset = "acs")

I tried and this fetch is actually pretty fast, even though it's
getting data for over 2,000 tracts.  And then pulling out the 591 you
want should be simple, since it's all on your own machine.

--Ezra 

On Tue, 19 Apr 2016 09:16:53 -0400, Michael Williams wrote:
> 
> [1  <multipart/alternative (7bit)>]
> [1.1  <text/plain; utf-8 (base64)>]
> [1.2  <text/html; utf-8 (base64)>]
> Hi Jonah,
> 
> I’ve used this package a fair amount. I also wrote my own very basic package for using the
> census api for slightly different use cases. Long story short, whenever possible I’d
> suggest using the wildcard as Carl suggests. That results in one api call while if you
> query each individual tract, you’re making a call for each tract and the data will take
> considerably longer to pull (with more opportunities for connection errors). The acs
> package is essentially looping through each geography, downloading, and binding the
> results together.
> 
> After you’ve got all the tracts from the wildcard you can filter the results locally.
> 
> Michael Williams | Financial Analyst | Community Development | Federal Reserve Bank of
> Minneapolis | 612.204.5572 | www.minneapolisfed.org
> 
> From: acs-r-bounces at mit.edu [mailto:acs-r-bounces at mit.edu] On Behalf Of Ganz, Carl
> Sent: Monday, April 18, 2016 7:43 PM
> To: Jonah Newman <jnewman at chicagoreporter.com>; acs-r at mit.edu
> Subject: Re: [acs-r] Error message in acs 2.0
> 
> Hello Jonah,
> 
> I have had similar problems when dealing with geographies with many elements. I am
> guessing that you generated your geo.set by passing a vector of desired census tracts
> rather than doing something like,
> 
> geo.set(state=”IL”, county=”Cook”, tract=”*”)
> 
> As a result, your geo.set list has 591 geo objects. From my experience, any geo.set list
> with more than 550 elements will generate the error you mentioned. I have no idea whether
> this is a design decision, a bug, or some other issue.
> 
> When I had this problem I divided my geo.set into two geo.sets, each with less than 550
> elements, fetched results for the two geographies, and then combined the output. Perhaps
> there is a more elegant workaround.
> 
> Kind Regards,
> 
> Carl Ganz
> 
> Assistant Statistician
> 
> UCLA Center for Health Policy Research
> 
> From: acs-r-bounces at mit.edu [mailto:acs-r-bounces at mit.edu] On Behalf Of Jonah Newman
> Sent: Monday, April 18, 2016 2:30 PM
> To: acs-r at mit.edu
> Subject: [acs-r] Error message in acs 2.0
> 
> Hi everyone,
> 
> I’m new to this listserv, and pretty new to R. I’ve used the acs package once before, on a
> fairly small set of geographies, and it was great! Now I’m trying it on a larger set, and
> it’s having some trouble. For reference, I’m using R version 3.2.5, and R Studio version
> 0.98.1103.
> 
> My geo.set has 591 elements (census tracts). I used check=T to make sure they were all
> good, and they are.
> 
> But when I try to use acs.fetch to download data, I’m getting the following error:
> 
> Error in if (url.test["statusMessage"] != "OK") { :
> 
>   missing value where TRUE/FALSE needed
> 
> Here’s my code:
> 
> #select race by Hispanic Origin data from ACS for selected geographies
> 
> race <- acs.fetch(endyear=2014, span=5, geography = mygeo, table.number="B03002", dataset
> = "acs")
> 
> Can anyone help me figure out what’s happening, and why?
> 
> Thanks!
> 
> Jonah
> 
> ---
> 
> Jonah Newman
> 
> Data & Investigative Reporter
> 
> The Chicago Reporter
> 
> (312) 673-3807 (office)
> 
> (312) 857-8675 (cell)
> 
> @jonahshai
> 
> The Chicago Reporter is a nonprofit investigative news organization that focuses on
> race, poverty and income inequality.
> 
> This e-mail message, including attachments, is for the sole use of the intended recipient
> (s) and may contain confidential or proprietary information. If you are not the intended
> recipient, immediately contact the sender by reply e-mail and destroy all copies of the
> original message.
> 
> 
> [2  <text/plain; us-ascii (7bit)>]
> _______________________________________________
> acs-r mailing list
> acs-r at mit.edu
> http://mailman.mit.edu/mailman/listinfo/acs-r

--
Ezra Haber Glenn, AICP
Department of Urban Studies and Planning
Massachusetts Institute of Technology
77 Massachusetts Ave., Room 7-337
Cambridge, MA 02139
eglenn at mit.edu 
http://dusp.mit.edu/faculty/ezra-glenn | http://eglenn.scripts.mit.edu/citystate/
617.253.2024 (w)
617.721.7131 (c)