| 06:00:27 | mdev: | hey, does anyone know an equivalent to map->save in mapscript that instead of saving, just returns the resultant mapfile that would have been saved? |
| 06:15:59 | jmckenna: | save to a different filename? |
| 06:16:32 | hobu: | mdev: no, therre is not a method to return the mapfile to a buffer |
| 06:20:14 | mdev: | so the only way would be to output to a temp file, read back in, and delete file? |
| 06:20:22 | hobu: | right now, yes |
| 06:20:30 | mdev: | ok |
| 06:20:37 | mdev: | tell me if this is a crazy idea... I want to make an md5 hash of the resulting mapfile from a requested map through mapscript and add that hash to a database ( along with the png file that map resulted in ) so if anyone requests a map that has already been requested, I can check the hashes and serve the previously rendered image rather than having to render the map all over again which is rather time consuming |
| 06:20:38 | hobu: | just write it to your OS's temp directory and you won't have to worry about cleaning it up |
| 06:21:34 | hobu: | I think in practice you'll find that the one or two map renders you save per user isn't going to be worth it |
| 06:21:47 | hobu: | if your data is that dynamic, change your approach |
| 06:22:04 | mdev: | I have a few thousand layers though... |
| 06:22:18 | mdev: | takes between 1-5 seconds to render on a fast machine |
| 06:22:26 | hobu: | use something like TileCache for the truly static layers and then use a client that can talk tiles or use the GDAL tile driver |
| 06:22:37 | mdev: | some maps will be used frequently but it leaves possibility for panning/zooming from the default map |
| 06:22:54 | hobu: | thousand layers? you're design is flawed : ) who can visualize 1000 layers in a single map. What is the value? |
| 06:23:07 | mdev: | well I just have a lot of cities and stuff |
| 06:23:12 | mdev: | each city point is a layer |
| 06:23:20 | hobu: | why? |
| 06:23:39 | mdev: | is that bad? |
| 06:23:49 | hobu: | you're forcing mapserver to take the cost of opening a layer up for each point. |
| 06:24:00 | hobu: | instead of having a single layer with all of the pointts in it |
| 06:24:00 | mdev: | should they just each be a class then? |
| 06:24:17 | nhv_: | try a cities layer that contains all city points |
| 06:24:31 | hobu: | yes. You should have all of the city points in a single layer and then uses CLASSes to adjust your visualization |
| 06:24:42 | hobu: | mapserver expects you're working this way, and it is optimized for this |
| 06:24:46 | mdev: | I didn't realize there was much efficiency difference |
| 06:24:49 | mdev: | between layers and classes |
| 06:24:55 | hobu: | it is not optimize for having one point in every layer ; ) |
| 06:25:08 | mdev: | still... with or without all the cities it only affects about .2 seconds out of 4 seconds rendering |
| 06:25:09 | nhv_: | layer requires reopening the data set |
| 06:25:15 | mdev: | seems like most of the time goes for rendering polygons |
| 06:25:23 | mdev: | ah ok |
| 06:25:27 | hobu: | rendering what polygons? |
| 06:25:32 | mdev: | the country outlines |
| 06:25:43 | hobu: | which country outlines, and at what scale? |
| 06:25:52 | mdev: | the world |
| 06:25:56 | mdev: | at various scales |
| 06:26:00 | mdev: | from entire world to a single conutry |
| 06:26:05 | mdev: | that's why it ranges 1-5 seconds |
| 06:26:06 | hobu: | and what format is the layer in, and are you using one layer per country as you are doing for cities? |
| 06:26:12 | mdev: | shapefile |
| 06:26:25 | mdev: | yeah, I think it is one layer per coutnry |
| 06:26:38 | nhv_: | have you run the shptree utility |
| 06:26:45 | mdev: | no, what's that |
| 06:26:53 | hobu: | nhv_: shptree is no good if each country is its own file |
| 06:27:00 | mdev: | oh, no |
| 06:27:02 | mdev: | it's a worldfile |
| 06:27:20 | hobu: | shapefiles do not have worldfiles |
| 06:27:39 | mdev: | i mean its a shape file with the entire world in it |
| 06:27:54 | hobu: | with no classification? |
| 06:28:00 | mdev: | I don't need high definition so I just zoom into specific countries as needed |
| 06:28:03 | mdev: | only need the outlines |
| 06:28:05 | nhv_: | http://mapserver.org/utilities/shptree.html |
| 06:28:06 | hobu: | and no labels? This should be very quick |
| 06:28:07 | sigabrt: | Title: shptree MapServer 5.4.2 documentation ( at mapserver.org ) |
| 06:28:11 | mdev: | there are labels |
| 06:28:16 | mdev: | and the countries get colored in |
| 06:28:19 | mdev: | individually |
| 06:28:44 | hobu: | have you read http://mapserver.org/optimization/index.html#optimization ? |
| 06:28:45 | sigabrt: | Title: Optimization MapServer 5.4.2 documentation ( at mapserver.org ) |
| 06:28:46 | mdev: | I'll try changing layers to classes and see if that speeds things up |
| 06:29:53 | mdev: | ah, so if you run shptree once, it will speed it up in general? kind of like an index? |
| 06:30:06 | hobu: | it is an index |
| 06:30:20 | mdev: | how come shapefiles don't normally come with one then? |
| 06:30:33 | hobu: | but if you're using the world_borders shapefile, there's only ~3k polys in there, and it should be quite fast without an index as long as your labeling isn't too bad |
| 06:30:44 | mdev: | oh |
| 06:30:48 | mdev: | I have a label for each |
| 06:30:51 | jmckenna: | i run shptree on all shapefiles no matter what. |
| 06:30:55 | mdev: | ok |
| 06:31:03 | hobu: | if you have lots of labels in such a way that mapserver has to do a lot of iteration to determine how to place them so they don't collide, you're going to have a big performance it |
| 06:31:42 | mdev: | well I can't do anything about that cause the labels I do need... but it's only like 300 labels ( pre cities ) |
| 06:32:05 | nhv_: | jmckenna ++ |
| 06:32:19 | mdev: | Also, I don't show all cities at once... I show more cities as you zoom in more, so it is probably never more than 100 cities at once |
| 06:32:28 | hobu: | qix index is pointless if you're drawing *all* of the shapes : ) |
| 06:32:38 | nhv_: | that is true |
| 06:32:57 | mdev: | but when I zoom in, the extent is less so then it would help, right? |
| 06:33:04 | jmckenna: | u bet |
| 06:33:26 | mdev: | ok, just made one |
| 06:33:29 | mdev: | let's see if that helps |
| 06:34:04 | mdev: | cut off about 0.001 seconds off, yay |
| 06:34:22 | mdev: | 0.05 on the second trial |
| 06:34:27 | mdev: | this is just showing europe |
| 06:34:30 | mdev: | from world shape |
| 06:34:33 | jmckenna: | in fact i did some ( unreleased ) tests with gigabyte roads files for benchmarking...shptree indexes made huge improvements ( in the early days of mapserver i used to split up the roads files into smaller chunks with shp2tile...no more ) |
| 06:35:26 | hobu: | mdev: are you running things with shp2img in debug mode so you can see your layer timings? |
| 06:35:36 | mdev: | I tried that before |
| 06:35:43 | mdev: | it only complained about missing qix |
| 06:35:44 | hobu: | what does it say? |
| 06:36:00 | jmckenna: | -map_debug 3 --> use that |
| 06:36:01 | hobu: | you need to turn up the debug level so you can see how long each layer is taking to render |
| 06:36:10 | mdev: | iok |
| 06:41:45 | mdev: | 0.006 for most layers |
| 06:42:17 | hobu: | get rid of the thousands of layers and collapse them into 1 |
| 06:42:29 | mdev: | ok, let's see if that takes care of it |
| 06:42:29 | mdev: | thanks |
| 06:42:35 | hobu: | 0.006 * 1000s, remember |
| 06:42:39 | mdev: | I'll get back to yas in a little bit and see how it works |
| 06:42:40 | mdev: | I get ya |
| 06:42:42 | nhv_: | and run shptree on that file |
| 06:42:47 | mdev: | already did |
| 06:42:53 | mdev: | thanks folks |
| 06:43:23 | nhv_: | run shptree on the 'collapsed into 1' file |
| 06:43:32 | mdev: | huh |
| 06:44:14 | nhv_: | maybe I am confused if already one file you should be all set |
| 06:44:23 | mdev: | ah ok, yeah it was just 1 file |
| 06:44:26 | mdev: | only about 3 mbs |
| 06:44:39 | mdev: | sorry, 6 |
| 06:47:08 | hobu: | http://pastebin.ca/ a result of one of your runs so we can see please |
| 06:47:09 | sigabrt: | Title: pastebin - Type, paste, share. ( at pastebin.ca ) |
| 06:47:32 | mdev: | of shp2img? |
| 06:57:23 | mdev: | WOW! |
| 06:57:34 | mdev: | getting rid of city points layers barely helped at all |
| 06:57:42 | mdev: | but getting rid of seperate layers per country REALLY helped! |
| 06:57:55 | mdev: | droped the render time from 1.3 seconds to .3 seconds |
| 06:57:56 | mdev: | incredible |
| 07:21:42 | jmckenna: | mdev: good work. |
| 07:22:28 | mdev: | the 4 second ones came down to .3 as well |
| 07:22:47 | mdev: | the strange thing is that the times go UP when I zoom in a lot rather than down |
| 07:23:21 | mdev: | like if you zoom into a country soo much that there is no detail but just the color of the country spans the width of the entire map |
| 07:23:24 | jmckenna: | i see this usually when labels start turning on...causing mapserver to work harder |
| 07:23:26 | mdev: | that takes longer than showing detail |
| 07:24:26 | mdev: | I can't believe having layers slowed it down this much... see I was going by the examples and in the examples they were making everything a seperate layer |
| 07:24:43 | mdev: | so I just followed that. I couldn't imagine that doing seperate layers would make it take more than 10x longer |
| 07:28:00 | nhv_: | which examples were those ? |
| 07:31:36 | mdev: | the mapfile examples |
| 07:31:38 | mdev: | with great lakes |
| 07:35:51 | nhv_: | ah right in http://mapserver.org/tutorial/example1-4-map.html#example1-4-map that is because the layers are of different 'type' |
| 07:35:52 | sigabrt: | Title: Example1-4.map MapServer 5.4.2 documentation ( at mapserver.org ) |
| 07:36:46 | mdev: | oh ok |
| 07:36:56 | mdev: | I guess I just assumed you should put different things in different layers |
| 07:37:02 | nhv_: | thanks for pointing that out though |
| 07:37:09 | mdev: | didn't realize it took drastically longer to render seperate layers than seperate classes |
| 07:37:27 | nhv_: | should probably mention that in tutorial |
| 07:37:27 | mdev: | I figured it is different ways of "naming things" for convenience |
| 07:37:33 | mdev: | yeah, that would be great |
| 07:37:35 | nhv_: | understood |
| 07:39:41 | mdev: | what happens if I declare features that are outside of the map extent |
| 07:39:51 | mdev: | does that waste processing time to a great extent? |
| 07:40:16 | mdev: | for instance, I have a growing list of 3000+ cities |
| 07:40:44 | mdev: | I have them ranked so when user zooms in, it shows more cities |
| 07:41:05 | mdev: | however, I'm loading cities for entire world into the mapfile and as you get closer, more unrelated cities outside of extent are added |
| 07:41:42 | mdev: | would it help to preprocess these cities and test whether they are within extent or not before loading them into mapserver, or is mapserver good at discarding things outside extent? |
| 07:42:40 | mdev: | BTW, the cities are just a point and label |
| 07:42:45 | mdev: | and symbol |
| 07:46:56 | nhv_: | this is esactly what running shptree does for you |
| 07:47:06 | mdev: | sorry, I should have clarified |
| 07:47:14 | mdev: | the cities are coming from somewhere else |
| 07:47:20 | mdev: | they are being loaded in with mapscript |
| 07:47:46 | nhv_: | well some kind of spatial indexing will help |
| 07:48:11 | mdev: | I'm thinking of just making a function that checks if lat/lon if city is within extent of map, then add to layer |
| 07:48:21 | mdev: | I used to add all cities to layer |
| 07:48:26 | nhv_: | you might experiment with making shapefiles fo different scales |
| 07:48:40 | nhv_: | then running shapetree over those files |
| 07:48:53 | mdev: | oh |
| 07:49:08 | nhv_: | eg dump whole world of cities to a shapefile for appropriate zoom levels |
| 07:49:21 | nhv_: | then use shptree to index those |
| 07:49:21 | mdev: | oh, I'd rather not do that |
| 07:49:25 | mdev: | cause cities are more dynamic |
| 07:49:39 | mdev: | I mean the list is growing |
| 07:49:42 | nhv_: | or use postgis to store your cities |
| 07:49:45 | mdev: | and users can add new cities |
| 07:50:29 | nhv_: | figuring out how to spatially index your data is key to any gis like operation |
| 07:51:05 | hobu: | always remember this about mapserver configuration: MapServer mapfile is written in such a way as to minimize typing ( it is expected that users will hand-edit mapfiles ) and reward laziness |
| 07:51:06 | nhv_: | each dataset has its own quirks |
| 07:51:47 | mdev: | yup |
| 07:56:15 | mdev: | ok, my own php pre-processing helped majorly |
| 07:56:29 | mdev: | basically: if feature in extent, add to map, else don't |
| 07:56:46 | mdev: | I didn't realize mapserver got so slowed down by adding features not in extent |
| 07:57:09 | mdev: | perhaps that's another mention you can add to the tutorial... that adding stuff you don't need/use slows it down a lot as well |
| 07:57:27 | : | * nhv_ is reminded of the truism that the fastest programs are those that do the least work :- ) |
| 07:57:46 | mdev: | I used to think that I'd just give it everything and let it use what it needs... I didn't realize it slows it down THAT much! |
| 07:58:18 | mdev: | I thought it would ignore stuff out of extent... was that really such a ridiculous assumption to make? |
| 07:58:55 | nhv_: | but it has to 'read' that data and determine if it is in the extent |
| 07:59:11 | mdev: | yes but that shouldn't slow it down THAT much |
| 07:59:32 | mdev: | I went from .2 seconds to 1 second by adding 2000 cities out of extent |
| 07:59:38 | nhv_: | depends on the quanity of data |
| 07:59:44 | mdev: | yet, to have php weed out the unnecessary ones was much quicker |
| 08:00:24 | mdev: | so I added that now, but once again, I didn't think that mapserver would get slowed down more than the little teenie bit required to read in a little more data |
| 08:00:28 | nhv_: | better yet to figure out how weed out bunches of data at once eg like what shptree does for shapefiles |
| 08:01:05 | nhv_: | the more data you have the more important that is |
| 08:01:44 | mdev: | definitely |
| 08:01:56 | mdev: | I just made the deadly assumption that mapserver would do that automatically |
| 08:03:09 | nhv_: | well it does but it uses 'brute force' unless using a datastore that can take advantage of some form of spatal indexing |
| 08:07:03 | mdev: | I got the resultant mapfile from my script down from 23,000 lines to 11,000 lines by getting rid of unnecessary cities beforehand |
| 08:07:34 | RLIMA: | Hello, maybe someone can give me a straight answer. Is or is not possible to load DWG with MapServer? |
| 08:08:26 | RLIMA: | I've seen a lot of things on the web |
| 08:08:46 | RLIMA: | some say that is possible with FME, but FME is not supported by GDAL |
| 08:08:54 | RLIMA: | so i'm a bit confused |
| 08:11:10 | jmckenna: | RLIMA: some points: |
| 08:11:23 | jmckenna: | - all vector supported formats are listed at http://www.gdal.org/ogr/ogr_formats.html |
| 08:11:24 | sigabrt: | Title: OGR Vector Formats ( at www.gdal.org ) |
| 08:11:45 | jmckenna: | - you might try the #gdal channel for the home of the experts |
| 08:11:57 | jmckenna: | - a DXF driver is coming: http://fwarmerdam.blogspot.com/2009/12/ogr-dxf-driver.html |
| 08:11:58 | sigabrt: | Title: Frank's Geo-Geeking: OGR DXF Driver ( at fwarmerdam.blogspot.com ) |
| 08:12:01 | jmckenna: | hope that helps |
| 08:16:45 | RLIMA: | jmckenna thanks a lot |
| 08:17:04 | jmckenna: | RLIMA: no problem. |
| 09:11:09 | mdev: | Hi, I was testing my configuration with shp2img and I noticed that the labels don't show up in resultant image |
| 09:11:30 | mdev: | turned out it's because "TEXT" was not set in the mapfile resulting from mapscript save |
| 09:11:53 | mdev: | I added text to a pointshape as follows: |
| 09:12:06 | mdev: | $pointShape->set( "text", "London" ); |
| 09:12:25 | mdev: | the feature appears correctly in mapfile but not the text parameter |
| 09:26:33 | jmckenna: | mdev: i think the correct function is settext( ) http://trac.osgeo.org/mapserver/ticket/3063 ...FYI i found this by searching HISTORY.TXT in the source http://svn.osgeo.org/mapserver/trunk/mapserver/ |
| 09:26:34 | sigabrt: | Title: #3063 ( Can not remove the TEXT property in the CLASS object ) - MapServer - Trac ( at trac.osgeo.org ) |
| 09:27:09 | jmckenna: | implemented in 5.6.0-beta3 ( as you also can see in the history ) |
| 09:27:33 | jmckenna: | hope that helps |
| 09:30:31 | mdev: | sorry, was away from computer, let me check that out |
| 09:30:32 | mdev: | thanks |
| 09:31:25 | mdev: | undefined |
| 09:31:39 | mdev: | Fatal error: Call to undefined method ms_shape_obj::settext( ) |
| 09:32:17 | jmckenna: | maybe i'm reading that ticket wrong. anyway now you know where to search. you can also ask questions in that ticket, or send a message to mapserver-dev. good luck. |
| 09:33:01 | mdev: | its weird cause set( "text", "x" ) works |
| 09:33:06 | mdev: | in the resultimg image |
| 09:33:13 | mdev: | but it doesn't get added to mapfile when save |
| 09:33:25 | mdev: | not the end of the world for me cause it works but messes up debugging |
| 09:33:44 | mdev: | http://mapserver.org/mapscript/php/index.html#shapeobj-class |
| 09:33:45 | sigabrt: | Title: PHP MapScript MapServer 5.4.2 documentation ( at mapserver.org ) |
| 09:34:00 | mdev: | according to mapscript doc, "text" is a member of shapeObj and there is no settext |
| 09:34:20 | aboudreault: | you need to use set( ) |
| 09:34:28 | mdev: | I do |
| 09:34:42 | mdev: | but then it doesn't get added to the feature when I save to mapfile |
| 09:34:45 | jmckenna: | and i'm referring to the ticket which mentions the class-settext. anyway good luck, you now know as much as i do |
| 09:34:45 | mdev: | from mapscript |
| 09:35:02 | mdev: | are you telling me to set the text for the class rather than the feature? |
| 09:35:10 | mdev: | let me try that and see if it helps |
| 09:35:51 | mdev: | no |
| 09:35:53 | mdev: | no way... |
| 09:37:03 | mdev: | hmm |
| 10:33:56 | mdev: | when working with php mapscript... why do I even need to have a mapfile to start? |
| 10:34:11 | mdev: | mapObj ms_newMapObj( string map_file_name ) requires a mapfile |
| 10:34:25 | mdev: | why can't I just set all parameters in mapscript bypassing a mapfile alltogether? |
| 10:51:59 | ue: | mdevmdev, try '' as map_file_name |
| 10:52:21 | bitnerd: | anyone know the limits of where you can use variable substitution? I am trying to cascade/reproject a WMS-T and use "wms_time" "%TIME%" in my metadata section right now to no avail |
| 11:11:09 | bitnerd: | ahhh, found a way.... thanks to old tschaub post on ml -- add TIME=%TIME% to connection string |
| 13:04:28 | jmckenna: | danmo: i have to step out for a bit, but i'm back later tonight..i just want to let you know in case you/we are planning on the 5.6.0 announcement tonight. |
| 13:05:23 | mdev: | 5.6.0 announcement?? |
| 13:05:40 | danmo: | I'm thinking of sending the announcement tonight, yes, but we're not branching docs yet anyway, right? |
| 13:06:04 | danmo: | So I don't think I am waiting on anything from you to send it, or am I missing something? |
| 13:06:19 | jmckenna: | i guess not. i mean i was wondering if we were going to work on it tonight. |
| 13:06:32 | jmckenna: | sorry daniel, i was worried you were waiting on me |
| 13:06:39 | jmckenna: | i worry too much |
| 13:06:40 | jmckenna: | ha |
| 13:06:57 | danmo: | nope... I sent the draft announcement last week and got no replies, so I assume we're good to go |
| 13:07:12 | jmckenna: | ok cool...didn't want to leave you hangin |
| 13:07:21 | danmo: | I've just been busy with other stuff today ( including xmas shopping ) |
| 13:07:40 | jmckenna: | cool. ha. understood. |
| 13:08:14 | mdev: | any xmas goodies in 5.6.0? Like bug 979 getting fixed? |
| 13:09:00 | jmckenna: | mdev: i actually have you the link to HISTORY.TXT today, containing a list of all goodies. consider it our early christmas present to you. |
| 13:09:10 | danmo: | I see 979 was fixed over a year ago |
| 13:09:19 | mdev: | oh |
| 13:09:24 | mdev: | docs say it's still pending: |
| 13:09:25 | mdev: | http://mapserver.org/mapscript/php/index.html#outputformatobj-class |
| 13:09:26 | sigabrt: | Title: PHP MapScript MapServer 5.4.2 documentation ( at mapserver.org ) |
| 13:09:34 | jmckenna: | ouch. that hurts hehe |
| 13:10:30 | danmo: | jmckenna: I just saw your ref to http://www.nwgeospatial.com/news/ on #osgeo ... hilarious |
| 13:10:31 | sigabrt: | Title: Northwest Geospatial GIS Solutions ( at www.nwgeospatial.com ) |
| 13:10:40 | mdev: | is 5.6.0 mostly bug fixes or new features? |
| 13:10:46 | danmo: | I had seen other variants of this clip before ( not GIS related ), but this one is the best |
| 13:10:58 | jmckenna: | danmo: ha thanks! i wanted to RT it but i wasn't sure if it may offend some |
| 13:11:22 | jmckenna: | yes i saw the actual movie in ottawa a few years ago |
| 13:11:35 | danmo: | depends if you have friends at ESRI? ; ) |
| 13:12:33 | jmckenna: | i am just hesitant publishing something with nazis. no comment lol |
| 13:12:54 | danmo: | mdev: http://n2.nabble.com/5-6-0-release-announcement-tt4137154.html#a4188713 |
| 13:12:55 | sigabrt: | Title: Nabble - 5.6.0 release announcement / Threaded ( at n2.nabble.com ) |
| 13:13:56 | mdev: | cool |
| 13:14:03 | mdev: | so is this considered a current or stable release? |
| 13:14:06 | jmckenna: | danmo: you didn't reply to my email about my truck...i was worried something was wrong when that happened : ) ha |
| 13:14:11 | danmo: | jmckenna: ha, I see... didn't think of it this way. ( I think the original post I saw about this was from Arnulf, but he may have included a warning about it ) |
| 13:14:22 | jmckenna: | ah true |
| 13:14:25 | danmo: | nope, email just stuck in the pile |
| 13:14:35 | jmckenna: | k understood |
| 13:14:52 | danmo: | actually, this clicking sound is called "casser des noisettes" in French... and the known fix is to use higher octane gas |
| 13:14:54 | danmo: | : ) |
| 13:15:08 | danmo: | happens most of the time when going uphill |
| 13:15:31 | danmo: | so I second your friend's advice : ) |
| 13:15:44 | jmckenna: | hahaahaha yup so i learned. yup yup. luckily this guy was a backyard mechanic so i didn't cost me a thing..it was kind of fun. |
| 13:15:59 | danmo: | the type that works for beers ; ) |
| 13:16:10 | jmckenna: | yes that type : ) |
| 13:16:53 | danmo: | gotta go... buhbyela |
| 13:17:00 | jmckenna: | buhbyela! |