| 04:33:20 | kjartanb: | when I try to compile Mapserver I get this error configure: error: Could not find gd.h or libgd.a/libgd.so in /usr/lib/. Make sure GD 2.0.16 or higher is compiled before calling configure. You may also get this error if you didn't specify the appropriate location for one of GD's dependencies ( freetype, libpng, libjpeg or libiconv ). |
| 04:34:03 | kjartanb: | but when I try to locate libgd.so it is in /usr/lib/ |
| 04:35:37 | kjartanb: | when I do gdlib-config --libs then I don't get the -liconv is that the problem? |
| 05:40:34 | : | <_rash7> which is better ? php mapscript or python mapscript??? |
| 05:43:04 | : | <_rash7> any ideas??? |
| 05:43:12 | : | <_rash7> which is better ? php mapscript or python mapscript??? |
| 05:44:01 | : | <_rash7> if i use python mapscript, can i get more performance than phpmapscript |
| 05:44:04 | : | <_rash7> ??? |
| 05:58:14 | : | <_rash7> anyhelp |
| 05:58:37 | : | <_rash7> anyone used python mapscript |
| 05:58:59 | : | <_rash7> i need to build a new web interface for mapserver |
| 05:59:09 | mdev: | I use php mapscript |
| 05:59:13 | mdev: | it works pretty good |
| 05:59:19 | : | <_rash7> I need to choose python or php mapscript |
| 05:59:44 | : | <_rash7> i think python is having more performance? |
| 06:00:17 | : | <_rash7> than php? |
| 06:00:46 | mdev: | not sure about performance difference |
| 06:00:54 | mdev: | but PHP is very fast and well optimized |
| 06:01:08 | mdev: | there is probably more help, docs, and info available for php as well |
| 06:03:37 | jmckenna: | that is a good point mdev made - PHP mapscript is very well supported by the MapServer devs. If you are choosing other mapscripts then be aware that they are not as well supported as PHP, currently. |
| 06:04:45 | : | <_rash71> ok |
| 06:05:12 | : | <_rash71> i am plannig to build an interface based on python mapscript |
| 06:05:36 | : | <_rash71> mdev, whats ur opinion about it? |
| 06:06:00 | mdev: | I haven't used python but php mapscript is working great for me |
| 06:06:04 | : | <_rash71> Is it be a sure wastage of time? |
| 06:06:17 | mdev: | there are lots of tutorials, wikis, and help available for php |
| 06:06:18 | : | <_rash71> what web interface are u using |
| 06:06:23 | : | <_rash71> ok |
| 06:06:39 | mdev: | I really don't know about they python stuff so I can't compare |
| 06:06:41 | : | <_rash71> i am attached to python than to php |
| 06:06:43 | mdev: | just know php is working great |
| 06:06:52 | : | <_rash71> ok |
| 06:07:08 | mdev: | if all the info/help weren't available I probably wouldn't have been able to do it |
| 06:07:12 | mdev: | even with a good knowledge of php |
| 06:07:22 | : | <_rash71> are u using web interface. |
| 06:07:24 | mdev: | so the language knowledge isn't enough, it's just a prerequisite |
| 06:07:31 | : | <_rash71> such as ka-map |
| 06:07:34 | mdev: | no |
| 06:07:37 | mdev: | I made my own |
| 06:07:44 | : | <_rash71> ok |
| 06:08:45 | : | <_rash71> did u use any framework ( php framework )? |
| 06:09:27 | mdev: | dunno |
| 06:09:31 | mdev: | I just hand coded all the php |
| 06:09:38 | : | <_rash71> ok |
| 06:09:58 | : | <_rash71> can u recommend any php framework? |
| 06:10:04 | mdev: | no |
| 06:10:05 | mdev: | I handcode |
| 06:10:11 | mdev: | so I don't know |
| 06:10:14 | : | <_rash71> ok |
| 06:10:33 | : | <_rash71> did u used pywps |
| 06:10:35 | : | <_rash71> ?? |
| 06:10:39 | mdev: | no |
| 06:10:50 | mdev: | I just used mapserver, mapscript, and custom hand coded php script |
| 06:11:08 | mdev: | used the php mapscript and mapserver tutorials to learn the system and immitate to get my code started |
| 06:11:13 | mdev: | then it all morphed into it's own thing |
| 08:42:56 | : | * FrankW proceeds to setup a fastcgi environment for mapserver locally, reviewing the howto as I go. |
| 10:39:31 | CIA-9: | pramsey * r9672 /trunk/mapserver/ ( HISTORY.TXT mapfile.c ): Make %substitutions% case insensitive ( #3250 ) |
| 12:03:16 | FrankW: | danmo: how much time are you willing to have spent on this ecw memory problem? |
| 12:03:33 | FrankW: | I'm nearly three hours into it now. |
| 12:03:41 | FrankW: | And by no means certain of resolving it. |
| 12:30:38 | danmo: | FrankW: sorry, I was away from my desk |
| 12:31:26 | danmo: | I saw your note in the ticket about when you let the process run up to 32MB of virtual memory... but do you have more than 32MB of RAM+swap on this system? |
| 12:31:36 | FrankW: | 32GB |
| 12:31:39 | FrankW: | Not MB. |
| 12:31:46 | danmo: | Sorry GB |
| 12:32:00 | FrankW: | I do not have 32GB of RAM+swap. |
| 12:32:14 | FrankW: | But on 64bit system unused virtual memory pages can be cheaply allocated to a process. |
| 12:32:33 | FrankW: | On a 32bit system I imagine there are no more than 4GB of such virtual pages available due to the small address space. |
| 12:32:35 | danmo: | how is it possible to allocate more virtual memory than you have available? |
| 12:33:06 | FrankW: | I memory page can be mapped into the address space without actually being used in RAM or swap. |
| 12:33:20 | danmo: | We are running this inside a OpenVZ environment, and when once we hit the memory limit the process crashes |
| 12:33:24 | FrankW: | But it does tie up some sort of VM management table entry. |
| 12:33:36 | FrankW: | ok, that is conclusive enough that it is a problem. |
| 12:34:00 | danmo: | aboudreault: can you please confirm this is what is happening? ( i.e. the crash when we allocate more than 8GB on the server? ) |
| 12:34:27 | FrankW: | I currently have a vague hope that a forced call to NCSThreadCleanInfo( ) at a key point may help the situation and am pursuing that. |
| 12:34:46 | FrankW: | But I'm hesitate to disappear down a rabbit hole, piling up time if the issue isn't too important to you. |
| 12:35:08 | danmo: | we can justify another 5 hours on this |
| 12:35:25 | FrankW: | Including the 3 I mentioned, or in addition to that? |
| 12:35:26 | danmo: | I consider this critical, but not only for us |
| 12:35:31 | danmo: | on top |
| 12:35:40 | jmckenna: | ticket number? |
| 12:35:44 | FrankW: | ok, I'll keep you guys posted via the ticket. |
| 12:35:56 | FrankW: | http://trac.osgeo.org/mapserver/ticket/3245 |
| 12:35:58 | danmo: | Am I missing something in thinking that this would be an issue for lots of GDAL/ECW users? |
| 12:35:59 | sigabrt: | Title: #3245 ( Raster layer with ecw data in fastCGI mode can fill up memory quickly ) - MapServer - Trac ( at trac.osgeo.org ) |
| 12:36:06 | jmckenna: | thanks |
| 12:36:18 | FrankW: | danmo: it only affects users that open and close ecw files a lot in one process run. |
| 12:36:36 | FrankW: | So it won't affect non-fastcgi mapserver, nor will it be a problem for typical desktop or commandline use. |
| 12:37:02 | FrankW: | and fastcgi-mapserver users who can afford to keep the files open ( only a few ecw files ) are also not affected. |
| 12:37:13 | danmo: | with a tileindex layer with > 100 images, we cannot afford to keep them all open |
| 12:37:33 | FrankW: | Undestood - but this is a relatively uncommon circumstance. |
| 12:38:20 | danmo: | don't want to argue unnecessarily, but how do people deal with serving large amounts ( TB ) of imagery data? |
| 12:38:40 | jmckenna: | agreed, this would be common |
| 12:38:43 | danmo: | I'm happy to use a better alternative |
| 12:38:53 | aboudreault: | I'm going to reproduce the problem on the dev machine.. but mostly sure that all crash because all the memory is used. ( that affect all other vitual machines ) |
| 12:39:28 | FrankW: | aboudreault: I suspect it fails because when the address space is exhausted new virtual pages can't be allocated and thread creation fails. |
| 12:39:56 | aboudreault: | That's in gdal code or ecw lib ? |
| 12:40:01 | FrankW: | I would expect you to see no significant impact on RAM or swap, but the process just blowing up when virtual memory gets too large. |
| 12:40:13 | FrankW: | I believe the issue is with the ecw sdk. |
| 12:40:57 | danmo: | Um... process blowing up is already bad enough |
| 12:41:03 | aboudreault: | On your machine, you can see that all memory is used but it doesn't broke anything? |
| 12:41:09 | FrankW: | danmo: yes, agreed. |
| 12:41:33 | FrankW: | aboudreault: on my machine "memory" is not used, though the virtual memory use of the process grows ( for instance past 32GB ). |
| 12:41:57 | FrankW: | But it isn't "using up" anything except address space and virtual memory page table entries in the OS which we can't normally see. |
| 12:42:14 | aboudreault: | Ok, in my virtual machines, I can see that "free -m" is growing too |
| 12:42:48 | FrankW: | I'm not familiar with this command. |
| 12:42:55 | : | * danmo still doesn't get the bit about memory pages being mapped but not used... gotta learn about that |
| 12:43:21 | aboudreault: | same for me. |
| 12:43:44 | FrankW: | I'm no whiz on how virtual memory works, but it is definately possible to "allocate" memory pages without ever touching them and making them appear in real RAM or swap. |
| 12:44:08 | FrankW: | Grr, my ecw sdk is not building. I have avoided it for a long time. |
| 12:44:32 | danmo: | you mean just malloc( ) without touching the actual memory, or using other function calls? |
| 12:44:54 | FrankW: | In this particular case the pthread_create( ) call is ultimately calling the mmap( ) sys call. |
| 12:45:04 | FrankW: | I was using strace to see so I don't know what intermediate functions are used. |
| 12:45:34 | danmo: | I never really played with mmap( )... time to read about it now |
| 12:46:26 | FrankW: | well, I need to run and pick up my son. |
| 12:46:28 | FrankW: | Later |
| 12:46:47 | danmo: | ttyl8r. thanks for looking into this and educating us : ) |
| 12:46:48 | aboudreault: | later |
| 12:47:34 | danmo: | aboudreault: is the VM 32 bits? |
| 12:52:38 | aboudreault: | danmo: no it's a 64bits vm |
| 12:53:20 | danmo: | ok, so if we crash after 8GB we are not hitting the limit of VM address space that FrankW mentioned and it must be something else |
| 12:53:43 | danmo: | or it's a limit specific to OpenVZ |
| 12:56:57 | aboudreault: | -bash: fork: Cannot allocate memory |
| 12:57:04 | aboudreault: | 3.5G |
| 12:59:28 | aboudreault: | though the mapserv process and apache haven't crashed. |
| 13:00:02 | aboudreault: | probably just because they hadn't the need to allow memory. |
| 14:01:29 | : | * tomkralidis is getting a chuckle out of http://n2.nabble.com/FW-MapServer-enhancements-refactoring-project-td2571268.html |
| 14:01:30 | sigabrt: | Title: Error 500 url=http://216.139.236.80/FW-MapServer-enhancements-refactoring-project-td2571268.html null agent ( at n2.nabble.com ) |
| 14:50:37 | aboudreault: | rewriting mapserver in java !?! |
| 14:50:43 | aboudreault: | *shrug* |
| 14:51:14 | aboudreault: | Hopefully it's just a thread ; ) |
| 16:04:59 | aboudreault: | FrankW: woot! : ) |
| 16:06:11 | aboudreault: | Will test that as soon as I'm at the office tomorrow. |
| 16:07:20 | FrankW: | excellent |
| 16:08:35 | aboudreault: | You seem to find that so quickly.... and it's in the ecw lib :/ |
| 16:09:45 | FrankW: | quickly? It didn't *feel* quick. |
| 16:11:11 | aboudreault: | Compared to what I would have take to do this fix in that code, it is. : ) |
| 16:11:31 | FrankW: | strace, and gdb were my friends. |
| 16:11:40 | FrankW: | But it was still a bit like shooting in the dark. |
| 16:12:02 | FrankW: | I also spent some time watching what was happening in /proc though that did not happen to help in this case. |
| 16:13:04 | aboudreault: | ok |