| 00:41:38 | : | * mauricio_ is away: Gone away for now |
| 00:58:48 | : | * mauricio_ is back. |
| 02:17:38 | : | * mauricio_ is away: Gone away for now |
| 02:46:38 | : | * mauricio_ is back. |
| 04:37:28 | : | * mauricio_ is away: Gone away for now |
| 04:40:38 | : | * mauricio_ is back. |
| 07:22:23 | marisn: | Hello. Is it just for me trac.osgeo.org sloooow? |
| 07:24:16 | wildintellect: | theres a good chance - the server team has been troubleshooting for a few days now |
| 07:24:22 | FrankW: | we are experiencing system problems. |
| 07:24:36 | FrankW: | I'm trying to take a bit of time to investigate before restarting apache. |
| 07:27:43 | marisn: | a memleak in Trac Python Apache stack? |
| 07:30:01 | FrankW: | marisn: that is unclear. It does not seem particularly like a memory leak. |
| 07:30:18 | FrankW: | I personally suspect we have a problem with our trac/postgres integration. |
| 07:30:41 | FrankW: | We have 205 postmaster processes running - presumably interfaces between 30+ trac databases and many httpd processes. |
| 07:30:53 | : | * marisn got strange OOM issue on one of Gentoo servers some weeks a go. No idea why. |
| 07:31:04 | marisn: | FrankW: what You use for Trac backend? |
| 07:31:18 | marisn: | I mean - Postgres? MySQL? |
| 07:31:23 | FrankW: | postgres |
| 07:35:59 | : | * marisn has no idea how to get usage stats from Postgres. pgstatsinfo? |
| 07:44:54 | FrankW: | marisn: osgeo.org services should be good again. |
| 07:45:58 | marisn: | still simply rebooting is not a longterm solution... |
| 07:46:23 | marisn: | yes. it flies. |
| 07:47:19 | FrankW: | No, this is the second time in a couple hours I've had to restart apache. |
| 07:47:32 | FrankW: | We are working on a migration strategy to new server( s ). |
| 07:47:49 | FrankW: | But I really think if we understand the issue with the current server the hardware would be sufficient. |
| 07:55:56 | marisn: | couple hours?!? this sounds really bad... |
| 07:56:26 | marisn: | try to rise trac log verbosity? |
| 07:59:58 | marisn: | also are trac logs clean? http://trac.edgewall.org/ticket/4465 |
| 08:00:00 | sigq: | Title: #4465 ( Timeout reached while waiting for a connection ) – The Trac Project ( at trac.edgewall.org ) |
| 08:00:30 | FrankW: | I'll check that. |
| 08:03:21 | marisn: | FrankW: I'm sorry for putting my nose in not mine business |
| 08:04:30 | FrankW: | hints are appreciated, though we have several folks digging into this for a while. |
| 08:05:32 | : | * marisn would start with software upgrade, check apache process count == db accepted connection count, and top : ) |
| 08:15:26 | TylerM: | boy, pidgin seems so unstable :( |
| 08:15:39 | TylerM: | or is it just when running with skype? |
| 08:15:50 | crschmidt: | The latter. |
| 08:16:47 | TylerM: | k thx.. will try again.. I was sure it couldn't suck so bad generally |
| 08:25:20 | pramsey: | FrankW: could put a pgpool in between the httpd and postgres backends |
| 08:25:41 | FrankW: | pramsey: care to explain further? |
| 08:25:44 | pramsey: | also, is the #of backends == #of apache frontends? or >? |
| 08:26:06 | FrankW: | I believe the number of backends ends up being #httpd * #tracinstances. |
| 08:26:11 | FrankW: | yes, bad arrangement! |
| 08:26:30 | pramsey: | so trac makes it's own connections... |
| 08:26:36 | pramsey: | are the trac connections persistent? |
| 08:27:20 | FrankW: | I believe they are persistent - presumably retained by the trac mod_python module. |
| 08:27:36 | marisn: | #tracinstances should be controlled by apache if frefork is used |
| 08:27:49 | marisn: | s/frefork/prefork |
| 08:27:52 | pramsey: | rather then getting a new backend for each connection, pgpool will operate as a pool, so you'd only have one connection per database |
| 08:27:58 | pramsey: | #tracinstances, presumably |
| 08:28:21 | FrankW: | that would be quite reasonable. |
| 08:28:31 | FrankW: | Is http://pgpool.projects.postgresql.org/ the right place to read about pgpool? |
| 08:28:32 | pramsey: | so all the tracs would connect to pgpool, and pgpool could connect to pgsql |
| 08:28:33 | sigq: | Title: pgpool main page ( at pgpool.projects.postgresql.org ) |
| 08:28:44 | pramsey: | yes |
| 08:29:34 | FrankW: | This wasn't really how I had planned to spend this day. :- ) |
| 08:30:32 | pramsey: | then don't, the world won't end if trac service is slow : ) |
| 08:31:04 | FrankW: | system functioning is significantly hampered. |
| 08:31:14 | FrankW: | And it makes www.osgeo.org, svn and trac nearly useless. |
| 08:34:49 | marisn: | pramsey: IMHO Trac by default uses permanent connections to Postgress. |
| 08:35:26 | marisn: | also being unable to report/search bugs is bad for hosted projects ; ) |
| 08:36:30 | pramsey: | well, that explains 200+ connections, and possibly the research exhaustion too |
| 08:36:45 | FrankW: | research exhaustion? |
| 08:36:50 | pramsey: | resource exhaustion |
| 08:36:56 | FrankW: | ah,yes. |
| 08:37:00 | pramsey: | I'm losing my ability to form correct words as I get older. |
| 08:37:20 | pramsey: | Increasingly my emails read like they are written by ESL students. |
| 08:37:25 | FrankW: | I am also suffering from some research exhaustion on this problem - I wasn't sure if you were making some sort of related joke. |
| 08:37:55 | pramsey: | 200+ * 8Mb in work memory == lots of memory used up |
| 08:37:58 | FrankW: | I am also often not a credit to my "english as a first language" status. |
| 08:38:46 | TylerM: | frankw - you think it's trac bogging down other http stuff then? |
| 08:39:03 | TylerM: | we could look at moving drupal over to osgeo2 |
| 08:39:11 | : | * TylerM is pretty getting ahead of himself again |
| 08:39:17 | FrankW: | I suspect the outragous number of postmaster processes is causing problems. |
| 08:39:17 | pramsey: | slow down, do the pgpool thing |
| 08:39:48 | TylerM: | if it means the site becomes useable, moving it would be worth it ; ) |
| 08:39:53 | marisn: | pramsey: will it help if for some reason old connections remain open ( are not properly closed )? |
| 08:39:53 | TylerM: | sure enough.. carry on ;- ) |
| 08:40:22 | pramsey: | if by help you mean hurt, yes. that's more resources used but not helping |
| 08:40:52 | FrankW: | I must confess I'm not getting an immediate sense of comfort from the pgpool web pages. |
| 08:41:26 | TylerM: | peer1 at 8am eastern: "We were able to login into your server and fix the backup's issue that was affecting it and backups are again running successfully." |
| 08:41:33 | marisn: | FrankW: how about httpd processes? also lot's of them? |
| 08:41:45 | pramsey: | http://pgfoundry.org/projects/pgpool/ |
| 08:41:46 | sigq: | Title: PgFoundry: pgpool: Project Info ( at pgfoundry.org ) |
| 08:41:56 | FrankW: | 17 httpd processes active at this moment. |
| 08:42:11 | FrankW: | TylerM: any progress on our ability to file a ticket with peer1? |
| 08:42:13 | marisn: | active or total count? |
| 08:42:24 | FrankW: | [root@osgeo1 ~]# ps -ael | grep httpd |wc |
| 08:42:24 | FrankW: | 17 238 1224 |
| 08:42:40 | FrankW: | you can also inspect http://trac.osgeo.org/server-status if you want a view of what's up. |
| 08:42:41 | sigq: | Title: Apache Status ( at trac.osgeo.org ) |
| 08:42:45 | FrankW: | Note that the server is well behaved right now. |
| 08:43:04 | FrankW: | wow, only 48 postmaster processes. |
| 08:43:35 | FrankW: | I'm starting to doubt my theory that connections are never closed off. |
| 08:43:56 | : | * FrankW sighs. |
| 08:45:28 | marisn: | any chances on upgrading trac? |
| 08:45:41 | FrankW: | that would be very tramatic. |
| 08:45:54 | FrankW: | we run an old old version with some local customization for ldap integration. |
| 08:46:09 | : | * marisn have read about that |
| 08:48:04 | FrankW: | TylerM: any answer on the peer1 issue? |
| 08:50:43 | TylerM: | FrankW: discussing it with them right now |
| 08:50:49 | FrankW: | cool, thanks |
| 08:51:01 | TylerM: | i was wondering if jason knew how to get in |
| 08:52:17 | TylerM: | the password email reminder tool didn't work for me - so they did it manually for me |
| 08:52:22 | TylerM: | back in a minute |
| 08:55:19 | : | * TylerM waits for the email... |
| 08:55:28 | TylerM: | i wonder how well postmaster is performing |
| 08:55:43 | TylerM: | ~yawn~ |
| 10:58:22 | TylerM: | http://www.agu.org/meetings/fm09/ down for you? |
| 10:59:01 | marisn: | works ( 1 kb/s ) |
| 10:59:05 | bitnerd: | TylerM, seems up but deadly slow |
| 10:59:13 | sigq: | Title: 2009 AGU Fall Meeting ( at www.agu.org ) |
| 10:59:47 | TylerM: | thx bitnerd |
| 11:00:12 | bitnerd: | sigq would seem to agree with two minutes to get the title |
| 14:24:53 | bitner: | cheers |