Register | Login
Attackpoint - performance and training tools for orienteering athletes

Discussion: Lidar for 133 US cities

in: Orienteering; General

Jul 15, 2019 5:38 PM # 
Spike:
This was new to me and might be of interest to some mappers:

ftp://rockyftp.cr.usgs.gov/vdelivery/Datasets/Staged/Elevation/Non_Standard_Contributed/NGA_US_Cities/

They've also included some vector data like building footprints, forested areas, and individual trees.

Check the 00_NGA 133 US Cities Data Disclaimer and Explanation Readme.pdf file for more details.
Advertisement  
Jul 16, 2019 4:27 PM # 
gordhun:
Okay, that is interesting. I'm most interested in the areas that have .las and .laz for download.
So I looked at one city that has an interesting park and there is a folder with a gazillion .las tiles.
Problem is -and there may be a simple explanation- how do we tell which tiles are the ones that apply to that particular park? Is there a map index for each place among those files.
Jul 16, 2019 4:47 PM # 
cedarcreek:
clickable link: FTP for NGA_US_Cities

@gordhun: It looks like the .las files are in the PC directory/folder. That probably means "Point Cloud". Look for another folder called "PC_Footprint". Download the .shp file and all the other files with the same first part of the name but different extensions. Put all the files in the same folder (together).

I open the shp file in QGIS using the "V", "Add Vector" button, but you might be able to open it in OCAD 11 Pro or higher. In QGIS, I then zoom to that layer, then use the Web menu to add something like OpenStreetMap. There are options for Google and Bing (map and satellite/aerial), but I find they don't work very well. When you use the weblayers, it automatically changes the CRS to match that needed for those weblayers (maybe 3587? I'm going off memory). Now put the weblayer under the shp file in the layers list. If the tiles are opaque, you can double click on the layer (to the left) and change the fill to be "no fill".

Then you zoom in on your area of interest (possibly hiding the shp grid for convenience, but turn it back on for the next step). Then click once on the shp layer to select it, select pointer tool (which might have an "i", for "information"), then click the tile (I start with upper left), and write down the tile name. Click on the lower right tile, and write that down. Then figure out the names of the other tiles. Sometimes it's an E-W and N-S numbering, sometimes it's just ordinal. Then use the ftp link to download those tiles.

If there are a lot, you can possibly make a list of full URL links, and use UGET to automatically download them. UGET is a separate download from the new "The National Map" Download site. It's not very intuitive, but it works gangbusters.

If you need QGIS in a UTM coordinate system afterwards, be sure to remove the weblayer, and re-set the CRS to your desired projection.
Jul 16, 2019 4:51 PM # 
igor_:
I just go to LAZ in Cloud and select an area and search for data and then click on metadata for tiles and it shows FTP link in there somewhere, and from there you can do what Matt says about figuring out tiles and I use wget to download tiles.

Here's an example metadata page -- has both LAS and LAZ links:
https://www.sciencebase.gov/catalog/item/5ad35c06e...

And for another area, just run this in your bash shell:
for x in 547 550 552; do for y in 797 800 802 805 807 810; do wget ftp://rockyftp.cr.usgs.gov/vdelivery/Datasets/Staged/Elevation/LPC/Projects/USGS_LPC_MI_31Co_Oceana_2016_LAS_2018/laz/USGS_LPC_MI_31Co_Oceana_2016_${x}${y}_LAS_2018.laz; done; done
Jul 16, 2019 5:25 PM # 
Spike:
@igor_ what's useful with the FTP data is that it includes data that is not currently in the national map. As an example, the FTP data includes the City of Dallas, TX. That's currently an empty spot for lidar from the national map.
Jul 16, 2019 7:55 PM # 
igor_:
Right, I guess they are running behind with indexing their data.
Jul 16, 2019 8:50 PM # 
hughmac⁴:
That data is incredibly rich. Thanks a lot!!
Jul 16, 2019 9:02 PM # 
gordhun:
CedarCreek I admire and envy your ability to navigate your way through the alphabet soup that is digital mapping. You are deftly driving a transport with 14 speed manual transmission.
I on the other hand am looking for the sedan automatic transmission version that involves importing laz and or las files directly in to the map via the OCAD Nw Map Wizard.
My go to source of LiDAR in the US is the National Map Viewer. It is hit or miss in two ways. 1) Sometimes there is no LiDAR available and 2) While they say all the information is based on the UTM coordinates that is not the case. Many projects in Florida and elsewhere are US State Plane and I recently came across an area in Virginia that used a third projection system, the name escapes me right now.

So Igor thank you very much for pointing out the Laz in Cloud page. It seems to be a more focussed version of the National Map Viewer site and lo and behold when I used it to search a state park in Florida where they want to have an orienteering map there was the LiDAR that I had not been able to find previously.

But where is the Download button??
Jul 16, 2019 9:29 PM # 
cedarcreek:
All I can say is QGIS is extremely useful when making basemaps and working with aerial photos,and what I've described is very easy stuff. It does appear to be opaque and difficult, but you have to start somewhere.

The most useful thing it does (IMO) is give you an easy-to-use front end for a lot of the really intimidating command-line only utilities like GDAL. You fill out a dialog box, and QGIS creates the command line for you. To me, this *is* the automatic transmission version.
Jul 17, 2019 4:22 AM # 
GuyO:
gord, cedar...
How available is lidar in Canada?
(I'm specifically thinking about Sommet Morin-Heights, the site of next week's junior camp)
Jul 17, 2019 3:04 PM # 
cedarcreek:
GuyO, I'm not sure about Quebec. New Brunswick has released all of their lidar online, I believe. A few years ago, I spent maybe four hours searching for lidar for downtown Calgary (parks along the Bow River), with no luck.

I usually start with a search that adds lidar after a place name, for the city, county, state, province, etc.

Where lidar isn't available free online, you can often walk into the "local" GIS office and try to play the non-profit card.

Another thing. This changes really fast, so you should repeat searches occasionally to see if anything has changed. I haven't heard of any national Canadian websites like the US's "The National Map" and "EarthExplorer".
Jul 17, 2019 3:35 PM # 
robplow:
Canada: It varies by province

Nova Scotia: https://nsgi.novascotia.ca/datalocator/elevation/

Manitoba: http://mli2.gov.mb.ca/dems/index.html

in both cases what is available on the website is not necessarily everything.

In Manitoba the City of Winnipeg data is not publicly available but we got hold of the bits we wanted by asking the right person.

In NS Halifax Regional Municipality has its own lidar here https://www.halifax.ca/home/open-data/open-data-li...

Some other NS municipalities also have lidar not on the Geonova site but often someone local was willing to provide the data.

Quebec: try this (hope you can read French) https://www.donneesquebec.ca/recherche/fr/dataset/...
Jul 17, 2019 4:56 PM # 
Canadian:
There is some available for free in Ontario as well - I have huge chunks of the provincial data.
Jul 17, 2019 5:00 PM # 
cedarcreek:
New Brunswick has complete lidar coverage available online:

https://geonb.snb.ca/li/
Jul 17, 2019 5:06 PM # 
robplow:
From experience - just because there is no lidar coverage according to a provincial website doesn't necessarily mean there is none. Ask around, check with local municipalities
Jul 17, 2019 8:20 PM # 
hughmac⁴:
Heya gordhun!

I was struggling with the same thing (where's the index), and the individual LAS footprint data wasn't in PC_Footprint (that's just the overall 'footprint' for the whole city), but in CITY/not_loaded/shapefiles/CITY_las_layout* ... in case that's useful for you!

Beyond that, cedarcreek's instructions work like a charm! Got just the goodies I need ... mwa-ha-hah! That building data is amazing!
Jul 17, 2019 9:03 PM # 
gordhun:
Right now I'm waiting on the decompression and import of some 20 LAZ files for a park in Florida. I'm suspecting the process has stalled on the 12th file. I may have to break the project down in to smaller numbers of packets. (Or learn how to drive a standard)
Jul 17, 2019 10:00 PM # 
hughmac⁴:
Cool! I like to thin with las2las before import, especially with big merged tiles, like:

las2las -i *.las -merged -o merged.laz -keep_random_fraction 0.2

That’s keeping 20% of points. That can really speed up imports / Kartapullautin.
Jul 17, 2019 11:17 PM # 
igor_:
I doubt it improves KP quality though.
Jul 18, 2019 1:12 AM # 
hughmac⁴:
Sure, it doesn’t improve quality. But whether it hurts it enough to matter ... depends on the use case.

Working on one now at that reduction, I’ll do it without as well, and we can compare.
Jul 18, 2019 5:47 PM # 
Terje Mathisen:
@hughmac: You really don't want to simply keep a 20% fraction, instead it would be much better to maintain a consistent density over the entire area, i.e. reducing to maybe 2-4 pulses per square meter.The best way to do this is probably to use lasthin which keeps a given number of points per grid cell, with each cell 0.5x0.5m you would get 4 points per square meter.
Jul 19, 2019 6:18 PM # 
hughmac⁴:
Makes sense. I’d say the biggest diff I see is not contours, which are good, but veg and cliff ‘density’. I’ll give that a go and see the affects.
Jul 20, 2019 1:46 AM # 
gruver:
Aside: On some trials round here I'm getting a stippled effect on the slope gradient, which looks artificial to me. Wondered whether my point density was insufficient, it's about 6/sq.m. But Terje above suggests this should be plenty. Am using OCAD, are the algorithms as good as KP?
Jul 20, 2019 7:51 PM # 
Jagge:
I prefer keeping random fraction. Vegetation layers results as returns (=more points there), if you just always keep same fixed number per grid cell you loose that information.
Jul 20, 2019 7:54 PM # 
Jagge:
stippled effect on the slope gradien

KP doesn't make any slope images.
Jul 20, 2019 8:41 PM # 
Terje Mathisen:
@Jagge: I actually prefer to always process everything I've got, even if it takes a bit more time: After all the increase is simply linear for vegetation, ground points are more or less the same even with more pulses.

If you think you must thin, then you need to thin pulses, not individual returns! I.e. assuming you have all your points in time order, you should have one or more returns from each laser pulse, and then you decide how many first returns to keep: Each time you pick one such first return (by random or otherwise), you automatically include any secondary returns from the same pulse.

The result of this should give much better statistical properties, i.e. getting the same results as from a scan done with lower pulse repetition rate.
Jul 21, 2019 12:42 PM # 
Jagge:
My limited experince is random is just fine and thinnig pulses like that may actually be worse. dropping out entire pulses (all points from a pulse) may remove too much locally. But sure better than fixed amount of points per grid. And depends on how much you have to thin and is interpretation algorithm processing as data as pulses (KP doesn't). And it is about not thinning too much, better stay on safe side here. And processing all sure is best. But for KP thinning down to 2pt/m2 is usually fine, output doesn't change much if you thin less or use full data (algorithms were designed for sparse data in mind).

Please login to add a message.