Discussion: Anyone from Nova Scotia on AP?

It looks like you guys can make alot of lidar maps. You need to be smart about picking areas because it seems theres a bit of a "green" problem haha.

Would anyone around halifax be interested in some lidar maps like these?







Obviously the last two aren't great for running on, but the first one is pretty good imo...

The first one looks very nice indeed! Are the flat areas lakes/ponds or marshland?

The latter two needs recalibration of the green, it might be a lot more runnable than it looks now.

I think so! The flat area should be marshes / lakes. But im afraid they are fairly impassible (very very slow) just from guessing. I would like to think that the green isn't that bad, but just by looking from the roads and satellite Im afraid this is a pretty accurate description of it.

I think field checking would be the best way to determine whether its worth recalibrating.

The base contours are amazing though so if you can find logged out areas then there could be lots and lots of nice maps due to all of the the free lidar. (looks like 80% of the province is covered)

Halifax has apparently been very good about making their LiDAR information available for several years new. Unfortunately the LiDAR tiles do not cover the whole municipality. For instance I've not been able to find LiDAR for the Peggy's Cove area in the southern part of the county. Peggy's Cove is famous for its lighthouse and its beauty. For this orienteer it has a patch of intricately detailed bare rock terrain between the highway and the ocean that would make for frustratingly intricate orienteering on a par with maps seen of Greenland, Iceland and looking somewhat similar but smaller than maps I recently saw fron Northern Norway.

Location of the last image

Like Terje wrote there may be calibration issues. 11 km north from the last green image it does not look too bad. And lidar is available.

Looking at this website, https://nsgi.novascotia.ca/datalocator/elevation/ , I couldn't find lidar in the second area. But I agree it would be a great map due to the runnability!



The first picture is from this area where I would definitely make a map, because its so close to the city and its sort of logged out making it pretty runnable...

Found a good area only 30min away from Halifax. This was all the lidar around it, so can't really make it bigger without more.



Link to google drive to download georeferenced file, and jpeg. You might need to ask me for permission.... don't know how to make it more public.

Im probably done with Nova Scotia until somebody from there says they are interested in maps....

https://drive.google.com/open?id=1LG_UexrqStrvrWVu...

tehre is older data not listed at that download site,
https://www.halifax.ca/home/open-data/open-data-li...
Just dem, but maybe data is available somewhere. If dem might be good enough for contour map.

Use batch mode to make the map seamless between tiles.

How do you use batch mode? Is there documentation or a tutorial somewhere?
I've been wondering about that....

This is a cut-and-paste of how to do batch mode in KP. I don't have time to reconsider/rewrite it right now:

To run a batch in KP, simply put all the lidar tiles into a directory \in under KP . I copy the "install" folder of KP with las2txt added into my project folder in a folder \KP. That has the pullauta.exe and pullauta.ini . Now create \in (in \KP) and paste the input tiles. Now open pullauta.ini in an editor like Notepad, and change batch process to 1. I typically leave number of processes as 1 unless you know what you are doing. My computer can handle 2 processes if the tiles are small, like 250m, or maybe 1km or 1 mile. As the tiles get bigger, you might have issues not because of processing memory (which is very moderate), but because the disk and memory during writes after the tile is complete. If the tiles are large, the batch will take a very long time and show very slow progress "status bars".

Edit: Also change "Save temp Files" to 1. You need the temp *files* to be able to merge the dxf files for vector import to OCAD or OOMapper. You do *not* need to save temp *folders* unless you are experimenting with those folders or something.

In the "install KP" folder that I paste into my project folder, I also create a DOS batch file KP_merge.bat with these three lines:

pullauta pngmergedepr 1
pullauta pngmergevege
pullauta dxfmerge

If you are using Mac, I don't know.

So when the batch process is finished, then just double-click that KP_merge.bat, and it will open a DOS window and merge images to create a merged_depr.png, a merged_vege.png, and several dxfs. I have a CSV I use to import the dxfs into OCAD.

Next, look at merged.depr.png. If there are errors of white spaces or obvious tiles that are only partly processed, you have to open the /out folder (which KP created), and type "depr png" into the search bar. Click View for large or extra large tiles (what ever is convenient). Look for those bad tiles. Here's the bad part. you have to delete the tilename.png *NOT* the tilename_depr.png . For some reason, Microsoft broke the regular expression part of that search bar, and it's a pain. Just be methodical and hopefully your tilenames are not stupid long and oh---the ones with NE/SE/NW/SW! I hate those.

Delete the pngs mentioned above, start pullauta.exe again (for batch mode, you just click it---you don't drag a laz file on top of it). This time, only do one process, and be sure to close anything taking resources. OCAD, Chrome, big images, etc. When it finishes, run KP_merge.bat again. Repeat until the merged_depr.png looks complete.

What the batch mode does is search for lidar point cloud data in adjacent tiles, so there should be *no* artifacts between tiles, only at the edges.

Now, let's go back. Say your data is big big, and it's in a state plane or something, When I tile using lastile above a threshold size, what I don't get is the intensity images that I create in OL-Laser, and our pro mapper asks me specifically for those a lot. Because of the lastools license, I often just do this part letting lastools strip the data, and then I do other heroic efforts to get a good intensity image for my pro mapper.

Here is how that works.

You can lastile without merging. Especially if you data is huge (you can select all tiles to see how big), just give up on merging.

So start by reprojecting into UTM using the original tile scheme. Oh---are you always using LAZ rather than LAS? If not, start now. LAZ is 1/7th the size of LAS, and it saves a ton of disk space.

So in your lastools command line, you can use -olaz to output in laz format. I often also use -odir and the folder like "-odir UTM", but you have to create the UTM folder manually---lastools won't do that.

Figuring out the projection command can be real hassle, but that's a whole other email. I process (reproject using las2las) the actual tiles from the download site, without merging.

So now you've got tiles of data with "wedges" but in meters and UTM (northern hemisphere).

Now lastile -i *.laz -olaz -odir in -tile_size 250

I use 250m in my KP /in folder because it's rocket fast with 2 process on my current computer (6GB RAM), and you can see it working with lots of action on the status bars. Once I get the KP batch output looking good, I delete the /in folder(s), but I've maintained the reprojected data somewhere. Then I'll tile at -tile_size 1000 for my OL-Laser run. I've never run with OCAD 12 or 2018, but you can experiment with a good tile size.

If you have enormous tiles, I typically run a clipping operation lasclip on either the merged laz file or on the reprojected tiled data. I only do this if I think the processing time benefit will be useful.

lasclip can clip to a shapefile polygon, but again, it strips that intensity that I like.

I leave task manager open all the time (possibly minimized to reduce resource use) so I can verify I'm not using too much CPU time, memory, or disk access.

I say it too late above, but seriously close down any resource hog while running lidar. A lot of problems are caused by resources.

Thanks alot for the super detailed instructions! It might be a while before I try going through these step by step, and also i think having a better computer could be helpful...

In short: make "in" subfolder and move your laser tiles there. In ini set batch mode 1. then execute it (double click the exe) and wait. And if you need vector output (not just rasters) set in ini save temp files to 1.

Oh crap! Yes! I forgot about the temp files! The merge won't work if you don't save temp files. You can merge the tilename_depr.png files in QGIS if you forgot, but you can't get the merged dxfs for vector import to OCAD or OOMapper.

Wow! That looks like a lot of work when OCAD 2019 will do most of that for you, just as OCAD 12 has been doing for several years.
You just open the DEM Import Wizard, select the laz or las files you wish to import, verify the projection system used for the file. If it is not UTM OCAD will automatically convert it to UTM for you then you sit back and wait for it all to happen. Along the way OCAD will ask if you want contours (original or smoothed) only or a product that comes out looking a lot like a Kartapaullatin map.
If time is money OCAD 2019 pays for itself in time saved in one or two maps a year.

KP contours are vastly superior, requiring less rework by the mapper.

It's also free.

Can I ask, if after I double clicked on the KP_merge.bat file and it made all the dxf files, I try opening the merged_depr.jpg and it says "it appears we don't support this file format" in Photos. Did I do something wrong, or how do I get around that?

How does KP get 'vastly superior' contours? The information comes from the same tiles.

There are many many ways to get contours out of lidar data. Lots of different algorithms that can give different outputs that are tuned for specific uses. KP does a really good job making contours that are useful for orienteering.

My experience is KP contours remove some of the contour detail that I want to retain in the final contours so I would not call them vastly superior as a sole source of automatically generated contours (i.e. if using KP-generated contours at all, I would also want another set of less smoothed contours generated using Terje's scripts and LAStools or OCAD 2019 to inform the final hand drawn contours). Perhaps this reflects, partly or entirely, a lack of nous on my part when it comes to choosing appropriate values for various contour generation parameters that can be specified in the KP configuration file.

Final contours on a map should almost always be hand drawn with the lidar only as reference.

I run a set of contours at 2m with heavy smoothing, and a set at 0.5m with minimal smoothing and have both visible for my fieldwork. The final contours I draw a mostly based on that 2m set, but having the noisier 0.5m set lets me add back in detail where needed, or use that detail for placing other features.

edwarddes, I'm assuming you use 2m with heavy smoothing when drawing two meter contours for sprint but not if doing 2.5 or 5 m contours?

I personally like to hand-draw all of my contours based on 5m and 0.5m contours and a hill shade and use the same for my field work (along with other layers for various vegetation information). I tend to generate the contours in OCAD (now using OCAD2019) but have used others on some projects.

Both KP and OCAD generated contours will always vary greatly from the final contours drawn by a professional mapper. At least in every example I've seen and compared.

Runner99 your area was probably too large and you resulted as jpg file with size 0. Check if can you open merged png file. If not, use a larger down scaling factor than 1 in your bat file, for example 2. Or simply open tiles one by one in qgis/ocad/oom and print it out from there.

"Both KP and OCAD generated contours will always vary greatly from the final contours drawn by a professional mapper."
I have found this varies with the terrain. For our gully spur terrain KP does a very good job and my hand drawn contours are quite close. But for mining terrain, I don't bother with KP contours at all.

TheInvisibleLog... good point I wasn't thinking of s
continental / spur fully terrain and have never mapped that kind of terrain myself. I could see how it would not need much adjusting from auto generated contours.

I am suffering terrain envy.

For those who are not familiar with KP and has't been following what it was written for and how has evolved over the years.

- KP was not written or intended for making base maps for O mapping. Some use KP's side products (vector contours or such) as base map, but the tool was not intended for it really.

- KP always does 2.5m contours (smoothed with interpretation) and 0.3 m contours (unsmoothed without any interpretation). It also makes third contours set (unsmoothed without any interpretation) with the custom interval user define is settings. Typically they seem to make 1.125m, 1m or 0.6126m contours for mapping. What mappers typically seem do do is use 2.5m auto interpretation contours as initial guestimate and for OOB parts, use custom interval contours when mapping and try digging details, shapes and accurate locations and such from 0.3 m contours when needed. No mapper I know does has tried to do mapping without any other control set then just 2.5m KP contours, that would not be smart thing to do because those had gone through some interpretation, simplification and smoothing algorithms to make them as usable as possible to be legible at 1:150 000 and to be navigated with out-of-the-box. So not quite something anyone should prefer to have as only contour set when mapping. On the other hand those side products, 0.3 m and custom interval raw unsmoothed contours may be just fine for mapping, but also not any better than contours made with other tools like OL laser, lastools or ocad. It is quite unfortunate if someone has tried to map mining terrain with KP 2.5 m contours without using at at least those 30cm contours for detail digging.

- so the whole point is making auto interpretation, getting at least some land form features mapped that normal algorithms fail to capture the same interval and also erase debris features to small to be mapped. Goal being producing something more suitable to be navigated with without any hand editing.

- even if those smooth spur gully terrains seem to suit well for KP it is not true, you don't need KP there, all you need to do is calculate regular contours and smooth them well you get equally good or better contours than KP ones.

- KP algorithms start to make difference in typical nordic terrain types with plenty of small knolls and such. These Halifax terrains should suit well. That does not mean contours produced here are better than in spur-gully or more ready to be used for regular O maps as they are, no, it is just the usability difference for navigation out-of-the-box compared to contours made with conventional methods is bigger in this terrain type than in spur-gully.

So is just all about out-of-the-box navigation use, not mapping.

Do how contours differ? Here is one example
https://routegadget.net/3d/pukkamaki/
gordhun, for example compare 2.5m contours of OL laser vs. KP and then see how pro mapper has drawn those parts that differ most.

So there is KP 2.5m contours, real O map made by professional mapper (not using KP), lastools and OL laser contours with various contours intervals. You can compare and see how KP contours are about halfway between raw and human made map, how it captured some stuff regular methods fail to capture and still erase debris contours, but failed to capture some and mapped some parts somewhat strange way. That should give impression why some consider them fine (and superior to raw conventional contours) to navigate with in these terrains straight away, but also why you should prefer more raw ones for mapping (when you start improving those you need to see raw ones to see what's actually going on).

Even if KP contours are often considered more usable for navigation than raw ones it is quite obvious any of you can quite easily make far far better contours than KP ones without going to forest just by making for example 0.25 contours and slope images, studying them and carefully armchair hand drawing final contours. And not just for mapping but also for navigation. The point for auto interpretation is making it possible to process larger areas. That sample area is just part of the larger area produced with KP, located here. If you zoom out you will soon notice it is too much work manually do the same area (~country). So those halfway to human mapper type contours are now here available thanks to processing effort Joakim and Mats did couple of years ago. But still if you start making maps here you better calculate raw contours for it.

And if someone is interested to see how things look at KP's home turf or is still not convinced KP 2.5m contours are fine for mapping without any additional raw contours, just go here and click the pink line at pointed location. And watch ten minutes how things look when I run that path though the terrain, compare map and scenery. You should see how lots of smaller this are missing from map - hints of those you might get from raw contours - but on the other hand pretty much all mapped can be spotted making it feel consistent and map is quite legible. So obviously something intended for navigation, but not mapping.

From my experience, here in Calgary we have like only 10ish decent forest maps. With I'd say 3 that are mapped well by professional mappers. The other 7 have been mapped a long time ago (1980-2000) and maybe by not the best mappers.

So currently if I had lidar of those other 7 areas, and ran it through KP then I would have contour maps that are way way more accurate then what I have now.

I think that also, if you use it in the right terrain, the contours will not change at all from what a normal mapper would do. It doesn't pick up rocks, small knolls, bare rock, and water features. But the vegitation areas and contours are quite accurate in my opinion.

What I think is its biggest strength is that it can spit out an orienteering map of a area (that can be very detailed) in a fraction of the time and cost that it would take to have it done by a professional mapper.

I believe that clubs should first start with lidar and KP maps, and once you have 50 maps around you, then invest in getting the best maps redone by a prof mapper. Until the mapper can finish the area then you can run / train on the lidar.

Once a finnish guy told me that they use KP / lidar to correct the hand drawn map contours. KP has no human error in my opinion, but every mapper maps differently.

Again I think in some terrains its perfect, in others, it might leave out shallow hills. But I've run on one KP map locally, and I thought the contours were more accurate then I could have made them myself. I'd just add a shallow hill or two here and there but 95% of the contours were accurate when I trained on it.





From these trainings I recall that the only place I'd change the map was at control 15, where it had picked up 3 small hills, but when I got there I saw 2 more. The thing is the other 2 might not have been 2.5m high, and maybe 2m tall, so thats also arguable, but I think that if its a hill it will usually be added regardless of the size.

When I was in Halden earlier, some of the knolls were so arguable (shallow), but I suppose that once you're used to what its mapped as then its easy to navigate by with consistency.

What I like about KP maps is that (so far what Ive experiences) if its on the map, then its 100% in the terrain and usually pretty obvious (if its a hill). If its shallow, it might have been missed out.

In my opinion I think KP is the future for starting orienteering in places with zero - very few maps, due to the economical result it has.

I think it just opens up so many possiblities when you can get this with zero effort:



and when the contours look like this:



Then I doubt that they would change much if a human remapped it.

That is some fun looking terrain!!

Runner 99: That is a very persuasive case for KP. I'm curious about a few things.
1) If you come across something that cries out for correction - a contour, perhaps or a yellow that should be blue how do you make the change?
2) How do you take off green that should not be green?
3) How do you place your courses on KP? I assume you use a second program such as purple pen?
4) Are there any capacity issues with KP? I ask because I recently tried to process 19 tiles of Ikm sq and either the computer or OCAD (64-bit) said that was too much for it so I had to break it into batches of 12 and 7 tiles.

@Runner99
It would be interesting to hear what maps you have today in the Calgary area. I was part of the team that made the first maps in 1980 with very poor base maps. I'm not really a mapper and wasn't then either but at that time better than the locals. The mapper on the team was Jan Samuelsson, you may have seen both his and my name on some 'recycled' maps.
Do you include maps like Bragg Creek and Bow Valley in your Calgary area?
The map at Bow Valley used for COC's in 1985 is certainly the best map I've run on in the area.

@gordhun
1) so after I made my maps and have the .pgw and the .png files, I'll import the .png into OOM and I think it georeferences and scales it automatically? idk how it works, but each time id import a tile they snap into place beside the other one. I also assume the scale is correct. Then I just use OOM to either make an OCAD file to use in Condes. Or if I just need a PTP course I use the course symbol in OOM and draw a course then export in the correct scale as a jpeg or pdf and then print that for my training.

I haven't made any changes yet to my lidar map (we only have one here), but I'm thinking I'd just put white forest or yellow over the area needing fixing and then correct the contours over top of that. Making just a tiny section of map. This takes very little effort and is not very time / money consuming. If you have imported the vector contours then you can straight up fix those.

2) do exactly the step above, but I'd say the better method is to fiddle with the settings until you have the green in fairly close shades that you want for the processing. If some green got chopped down and you have the vector contours, then just slap some white over top the green and its gone. If you don't have the vector contours you'd need to draw them over the section of white

3) after having a OOM file I can make it into OCAD for Condes, or just slap it into purple pen, or just use the course symbol as explained above.

4) I processed one chunk of 14km^2 in about 7-8hrs on my Dell Inspiron 13 laptop with i5 processor, and 8 GB RAM.

My computer can handle any 1 - 1.5km tile in 10-30min max depending on how detailed it is and how many knolls there are for it to pick up.

I've only batch processed those 6 Nova Scotia tiles and they took about 40min ish but idk how big they were they were rly fast.

I also think there is a mode where you can get only the png and pgw files and it goes waaay faster. Processes a tile in like a min or two.

For more tiles Im thinking of getting a better computer for the batch processing. 19 tiles of 1km^2 should be easy for KP and a decent computer. Less than a day. Im planning on doing much much larger areas, and with a better comp i think it'll be fine.

Also the trick is to pick areas that consist of mostly just contours and cliffs and vegetation. If there is lots of rocks or water features they'll be missing on the map and need to be added. But like Bow Valley is only contours and vegetation and thus is easy to get a very good map for from KP.

@bubo

Of the top of my head maps around Calgary and their quality:

W - worse than KP, B - better than KP, KP = KP

B:
-Bow Valley ISOM 2011 by Czech mappers. (waaay better than the old version)
-Aspen Creek ISOM 2013ish Remo Madella
-Rafter 6 (Ti chura Bi something) 2011 - 2016ish ISOM pretty good, contours wierd in some places, wish I could compare to lidar

-Canmore Nordic center (boring map, too many trails not enough forest)
-Quarry lake (new but like not mapped great, I don't like it but its probably just boring)

W:
- Elbow Bluffs (mostly useless super boring map, on part lower down is good, but like the hand drawn contours and veg is worse than KP imo)
- Connop creek, (half done map, not great sketchy in some places and fairly easy overall, KP would eb better)
-one I forget, where we just had AB champs, (very meh map)
-Mt Laurie (amazing terrain, countors are very old and kinda messed up. needs KP asap)
-Moose Creek (I'd give it like a 70%, KP contours and veg would be better but its still usable) club forgot about the map for like 10 or so years

KP:
West brag creek, map i posted above. All contours are super accurate, veg is like 80-90% good depending where u are. One part has interesting terrain. Only missing a few marshes but easy to orienteer without.

I'll omit mort of our park maps because they suck for training and i don't care about them. Lots are old too.

I mapped NoseHill actually by hand without lidar and its like pretty decent. I made it over detailed so I can read alot on trainings, but its not as good as West Bragg Creek. It is good that I can do like a 4:20min/km orienteering pace though so its fast. if you look through my imgur you can probably find alot of these maps. The vegitation would have been better if i had lidar though and KP.

Runner 99: Have you got access to LiDAR for the Crowsnest Pass/ Blairmore area? That area stands out as an awesome area for orienteering. (And the Canadian distributor for Silva products lives in Blairmore).
Produce a rogaine like KP map of that area and you can pretty well be certain that the AR community there will be using it.

With Bow Valley I guess I meant the map you call Mt Laurie (memory is not the best, long time ago...) and I agree on the terrain.

Hey Bubo: we're getting pretty far from the Nova Scotia topic but remember that 1985 event at Mt Laurie? Spectacular terrain but that year it was so dry that a couple of the best route choices were right across dry lake beds marked blue on the map.
We had a lot of Swedes participating that year. Somewhere I have a videotape of many of them coming out of the woods on to a power line and trail crossing and just keeping on going while pretty well all the Canadians and Americans came out of the same woods and on to the same trail and power line and stopped to check their map. Very telling that and things haven't changed much in the last 34 years.

@gordhun

I always wanted a map of the Frank Slide area, I think it would be really cool. I'd try and get it done by a good mapper though to add all the cliffs and rock features. KP would be a great start and would cover most of the ground contours there.

AB is like the worst province with lidar, I think its all avaliable but its $400 for 1km^2 and Im not gonna pay that out of my pocket especially if I don't have a job.

Currently not many in the club know about lidar and I'm trying to promote it as much as possible with KP to create initiative so that we can either get some for free by lobbying as a club, or just put aside some money in order to do it.

As there is no free lidar that I know of, I can't make any KP maps especially down by crowsnest pass which is 4 hours away.

Once we start moving in the right direction then I think we'll eventually make a map down there. But its just better for FWOC to develop areas an hour away first until we have enough of those.

runner 99: I did make a map of the Blairmore area using Google Earth and Canadian topographic images and it wasn't too bad but the contour interval was something like 10 meters.
Blairmore would make a good sprint venue and the foothills of their ski area, the area around the golf course and the XC ski area look to have great potential but 10 m contours just don't cut it. Now that I think of it perhaps they were 20 or 40 meters.
When AOA could not or would not be any help with LiDAR I shelved the Crowsnest projects. I don't blame them. The Province of Alberta seems to be bucking a world-wide trend to make LiDAR more readily and freely available.
And I agree with you to not go in to your own pocket to find money for mapping.

might be nice to see the map if you still have it. Can stop by if Im ever down there. Im hoping that in time AB will get free lidar. Until then paying for it is still ok for certain mapping projects. I know we have another bunch of it that is being used for a map we'll use for WCOC or something. I can't get the lidar until after WCOC's but once I do I'll grab it and KP it and we can compare it to the real map...

I just heard the area is not stellar but if its big enough I might get lucky with some decent contours somewhere....

Send me your e-mail address and I'll send you the Blairmore map file. gordhun at rogers dot com
I should suggest to Jim Webster that people driving from Calgary to next year's COCs stop by in Blairmore for a little R&O (rest and orienteering)

Has the club or provincial association tried asking whoever owns the lidar to waive (or significantly) reduce the fee? I would guess the reasoning behind the fee is that commercial organizations should be paying for data they can use to generate profits. The intention is not to hit not-for-profits.

Write a letter pointing out that orienteering is a small sport, run by volunteers on a not-for-profit basis and receives little or no government funding. Tell them how great an activity orienteering is that contributes to greater public well-being and include some maps of urban parks to show how useful O maps can be for the broader community. Etc etc. Maybe get a supporting letter from O Canada. Give examples of other provinces that make the data available for free.

It is worth a try

I sent an email today to an exec on the AOA board. I talked to them about it earlier (they were open to it) and thought I should give them a reminder today haha. I'd like if they wrote the letter, because they're in a better position to speak for the club and sport on a provincial level.

Yeah thanks robplow, i think thats the best move that we can do, hopefully if we do it right we'll get some good lidar from it either free or at a low cost!

Definitely the letter should come from the association - but you can make suggestions about what to write. Send them a suggested draft - the more you do for them the quicker things will happen.

Also ask around - does anyone know anyone who works in that field in Alberta? Try to find out who is in charge - a personal connection is always going to be more likely to succeed than a letter that just lands somewhere in the bureaucracy.