Discussion: Pullautin on Linux

Hi Jagge:

In one of your recent Pullautin versions (27-10-2012?), you (I presume accidentally) included your perl. One of the big troubles with the .exe has been the 32-bit memory constraints, and also that I have a big, fat, linux cluster just sitting here, with which to mega-batch process (maybe the entire Philadelphia Metro Area?).

Any chance I can get the latest source code, or just the pullautin.pl?

Thanks, and that fat ultra-long map is unbelievable! :)

-Hugh (hughmac4)

Why dont you just batch process lots in tiles (lots of laz files in same directory). People has processed a lot bigger areas than that ultra long map, about 5..6 times bigger than that, ~ 1500 km2. Having just one large chunk only makes the process slow, you can use just one core and you need lots of memory - not smart way to make large area. Instead copy paste several pullautin folders/instances with same input/output folder in in, and start them all. Uses 100% cpu and lots of memory without hitting 32bit memory constraint. batch mode makes and clips overlap between tiles, so there is no any edge between tiles.

P.S. -- I just processed 15 PAMAP (Pennsylvania MAP 1 mile sq. Tiles) LIDAR files in parallel. I can do more than that, but shouldn't abuse these work resources. :) Looks like a whole tile completely processed averaging about an hour -- x15 = ~39sq km in an hour -- crazy!

Linux Parallel Mapping Experiment

Ah, interesting the 'batch mode'. May I have the latest pullautin.pl file that supports batch mode (the one I have doesn't), and then I can do batch mode in parallel? The version I have is just a few days back from that. :) So if I start multiple pullautin processes on the same folder of LAS files, they keep track of which pullauta is doing what?

There should be a Karttapullautin Challenge for most km2/hr (or /day, or whatever). Give me the Perl, I want to win! :)

My pretty regular windows 7 laptop does about 35 km2 per hour. 4 core i7, 6 parallel processes processing 3x3 km tile each. This Finnish data may be lighter than yours, but on the other hand 3x3 tile is slower than 1 mile tile and my process does the overlapping/clipping etc so part of the are is processed twice.

Current batch mode doesnt store dxf etc data, just png files, so you must wait until I add that if you want the get that done. BTW, keep on experimenting, I enjoy following it!

Anyone convert the Maastotietoakanta import process to OpenStreetMap process? Before I dive into the code to see if it's even possible (don't see why not!) ... It's the only consistent data that I can get my hands on for most areas around here, and it would be a nice addition to your tool! :)

Just did some batch processed Finnish stuff, and am hooked on that process! Wow!

Oh, and did 40sq miles (~100sq/km) today in 1.6 hours. Would have been 1.3, but one tile was very complex.

Oops, only 38sq/mi, looks like I killed two. :)

Philly Area