Got error 28 from storage engine

As it turns out on a shared server after a scheduled backup at 2AM in the morning the /tmp folder becomes full and causes this error.

The server admin told me that this happens about once a week at CrystalTech. He said it is mostly due to some screwed up database and after the backup occurs the temporary folder becomes full.

I never ask but I am sure he is referring to the Coldfusion tmp folder because on my SolarDave.com site where it happened this morning PHP WordPress pages worked fine but only the Coldfusion pages pulled an error on more complicated queries. (happened only on my city CF pages and not the state pages)

2010-01-08 17:35:38.651 UTC
2010-01-08 17:14:21.481 UTC 20 minutes Communications link failure due to underlying exception:

2010-01-05 18:40:15.881 UTC
2010-01-05 18:46:10.473 UTC 6 minutes

2010-01-04 16:34:11.517 UTC
2010-01-04 16:29:09.396 UTC 5 minutes

2010-01-04 03:11:48.488 UTC
2010-01-04 03:06:15.093 UTC 5 minutes

2009-12-20 23:35:29.324 UTC
2009-12-20 23:37:48.407 UTC 2 minutes

December 05, 2009 / 07:39:48 AM
December 05, 2009 / 07:40:06 AM 1 minute unable to create new native thread null

2009-11-24 15:51:16.555 UTC
2009-11-24 15:49:15.626 UTC 2 minutes

2009-11-24 16:21:45.776 UTC
2009-11-24 16:17:20.589 UTC 4 minutes

2009-11-22 23:30:38.849 UTC
2009-11-22 23:27:22.315 UTC 3 minutes

2009-11-18 04:32:04.932 UTC
2009-11-18 04:33:19.198 UTC 1 minute

2009-11-10 14:34:36.47 UTC
2009-11-10 14:30:30.641 UTC 4 minutes

2009-11-11 15:58:58.508 UTC
2009-11-11 16:08:13.666 UTC 10 minutes router between db issues

2009-11-11 15:37:12.471 UTC
2009-11-11 15:49:43.179 UTC 12 minutes

2009-11-12 10:39:59.237 UTC
2009-11-12 10:38:35.524 UTC 1 minute

2009-11-12 09:09:52.568 UTC
2009-11-12 09:07:34.541 UTC 2 minutes

2009-11-09 00:40:36.53 UTC
2009-11-09 00:34:40.766 UTC 6 minutes

2009-11-02 22:45:29.976 UTC
2009-11-02 22:42:38.287 UTC 3 minutes

2009-10-26 11:33:32.018 UTC
2009-10-26 11:28:02.219 UTC 5 minutes

2009-10-22 22:46:03.255 UTC
2009-10-22 22:42:16.645 UTC 4 minutes

2009-10-19 22:43:46.829 UTC
2009-10-19 22:38:48.682 UTC 5 minutes

2009-10-17 20:11:19.655 UTC
2009-10-17 20:14:56.157 UTC 3 minutes

2009-10-16 22:53:42.282 UTC
2009-10-16 22:49:10.076 UTC 4 minutes

2009-10-15 22:40:34.021 UTC
2009-10-15 22:42:46.715 UTC 2 minutes

2009-10-15 03:02:03.901 UTC
2009-10-15 02:58:30.367 UTC 2 minutes (DB fail)

2009-10-13 17:46:44.918 UTC
2009-10-13 17:50:16.872 UTC 4 minutes

2009-10-12 15:37:49.005 UTC
2009-10-12 15:41:00.32 UTC 5 minutes

2009-10-09 09:29:27.209 UTC
2009-10-09 10:04:58.018 UTC 35 minutes

2009-10-08 15:55:41.352 UTC
2009-10-08 15:54:04.157 UTC 1 minute

October 07, 2009 / 08:26:32 PM
October 07, 2009 / 08:51:39 PM 25 minutes DB issue

2009-10-07 16:26:13.506 UTC
2009-10-07 16:25:54.717 UTC 1 minute

2009-10-03 19:46:52.205 UTC
2009-10-03 20:10:13.145 UTC 25 minutes. IIS crashed, but just my site was down. He said CF had issues before IIS crapped out

2009-09-30 14:11:14.856 UTC
2009-09-30 14:05:46.65 UTC 6 minutes

2009-09-24 12:43:11.456 UTC
2009-09-24 12:41:51.421 UTC 2 minutes

2009-09-17 00:50:27.019 UTC
2009-09-17 00:44:12.359 UTC 6 minutes

2009-09-09 15:15:45.815 UTC
2009-09-09 15:10:36.803 UTC 5 minutes

2009-09-09 14:04:32.526 UTC
2009-09-09 14:11:06.586 UTC 3 minutes

———————————————-
webE41/mysqlC1

2009-09-05 09:08:29.654 UTC
2009-09-05 09:05:36.23 UTC 3 minutes

2009-09-05 08:04:13.035 UTC
2009-09-05 07:59:32.838 UTC 5 minutes

2009-09-05 07:09:52.889 UTC
2009-09-05 07:00:57.1 UTC 9 minutes

2009-09-05 06:05:42.655 UTC
2009-09-05 06:01:50.785 UTC 4 minutes

2009-09-05 05:05:59.487 UTC
2009-09-05 05:00:06.389 UTC 5 minutes

2009-09-04 14:31:08.601 UTC
2009-09-04 14:26:21.821 UTC 5 minutes

2009-09-04 12:34:47.773 UTC
2009-09-04 12:24:52.471 UTC 10 minutes

2009-09-04 10:31:35.672 UTC
2009-09-04 10:26:52.738 UTC 5 minutes

2009-09-04 08:48:18.147 UTC
2009-09-04 08:33:33.295 UTC 15 minutes

2009-09-04 07:40:17.954 UTC
2009-09-04 07:27:45.273 UTC 13 minutes

2009-09-04 06:41:46.367 UTC
2009-09-04 06:26:58.042 UTC 15 minutes

September 03, 2009 / 02:38:06 PM
September 03, 2009 / 02:24:55 PM 4 minutes

September 03, 2009 / 11:04:52 AM
September 03, 2009 / 11:03:46 AM 1 minute

September 03, 2009 / 08:42:33 AM
September 03, 2009 / 08:42:32 AM

September 03, 2009 / 08:22:39 AM
September 03, 2009 / 08:18:21 AM 4 minutes

2009-09-02 00:39:35.32 UTC
2009-09-02 00:34:42.778 UTC 5 minutes (switched over)

2009-08-30 18:24:43.421 UTC
2009-08-30 18:30:17.154 UTC 6 minutes (switched over)

2009-08-29 16:53:29.777 UTC
2009-08-29 16:41:46.928 UTC 12 minutes (switched over)

August 28, 2009 / 09:51:27 AM
August 28, 2009 / 09:49:22 AM 2 minutes (switched over)

———————————————-
webE36/mysqlC1

August 26, 2009 / 08:00:07 PM
August 26, 2009 / 08:02:34 PM 2 minutes

August 26, 2009 / 04:28:16 PM
August 26, 2009 / 04:30:02 PM 2 minutes

August 26, 2009 / 03:06:30 PM
August 26, 2009 / 03:04:46 PM 2 minutes

August 26, 2009 / 01:55:12 PM
August 26, 2009 / 01:53:09 PM 2 minutes

August 26, 2009 / 11:16:43 AM
August 26, 2009 / 11:12:25 AM 4 minutes

August 26, 2009 / 12:17:56 PM
August 26, 2009 / 12:14:27 PM 3 minutes no switch over

———————————————-
webE36/mysqlb15

August 25, 2009 / 04:00:54 PM 1 minute
August 25, 2009 / 03:59:29 PM

August 25, 2009 / 02:53:13 PM

August 25, 2009 / 02:02:59 PM
August 25, 2009 / 02:03:43 PM 1 minute

August 25, 2009 / 01:50:08 PM 1 minute

August 25, 2009 / 12:37:33 PM 2 minutes
August 25, 2009 / 12:39:45 PM

August 25, 2009 / 11:46:56 AM 1 minute

August 25, 2009 / 12:03:16 PM 1 minute

August 25, 2009 / 09:03:51 AM
August 25, 2009 / 09:01:57 AM 2 minutes

August 24, 2009 / 12:43:36 PM
August 24, 2009 / 04:06:14 PM 7 minutes (3 separate times)

August 19, 2009 / 09:29:15 AM ***3 hours and 25 minutes Power outage on UPS all DB down (switched over)
8/19/09 12:52 AM

August 17, 2009 / 09:20:10 AM
August 17, 2009 / 09:58:00 AM 18 minutes total many outages during the work day (switched over twice)

August 15, 2009 / 01:04:14 PM

August 10, 2009 / 01:32:30 PM

2009-08-04 19:53:01.77 UTC
2009-08-04 19:50:59.589 UTC 2 minutes

August 04, 2009 / 11:01:23 AM
August 04, 2009 / 10:54:07 AM 7 minutes  (did switch over but after 6 minutes)

2009-07-29 18:39:12.243 UTC
2009-07-29 18:32:45.412 UTC 7 minutes (did switch over)

July 22, 2009 / 02:09:03 PM
July 22, 2009 / 02:04:15 PM 4:45 minutes (switch over didn’t work, ck log files)

2009-07-08 01:54:01.258 UTC
2009-07-08 01:57:47.521 UTC 3 minutes

July 01, 2009 / 09:18:08 AM

2009-06-16 18:40:49.733 UTC
2009-06-16 18:35:28.215 UTC 7 minutes

June 10, 2009 / 04:17:02 AM

2009-06-01 19:51:30.543 UTC
2009-06-01 19:58:47.698 UTC 7 minutes

May 27, 2009 / 10:25:23 PM 10 minutes (switch over didn’t work, PHP pages worked fine)
May 27, 2009 / 10:35:10 PM

May 26, 2009 / 07:37:10 AM

May 18, 2009 / 11:15:25 AM
May 18, 2009 / 11:09:44 AM 6 minutes

2009-05-11 20:48:18.177 UTC
2009-05-11 20:39:41.795 UTC 9 minutes

May 01, 2009 / 11:18:22 AM
May 01, 2009 / 11:14:33 AM 4 minutes

2009-04-30 19:26:36.755 UTC
2009-04-30 19:31:57.105 UTC 5 minutes

2009-04-28 16:11:41.351 UTC
2009-04-28 16:09:02.185 UTC 3 minutes

2009-04-20 21:06:30.478 UTC
2009-04-20 21:02:47.01 UTC 4 minutes

April 08, 2009 / 09:10:11 AM
April 08, 2009 / 09:08:59 AM 1 minutes

March 24, 2009 / 07:11:14 PM

March 17, 2009 / 01:33:15 PM
March 17, 2009 / 01:32:16 PM 1 minute

2009-03-06 06:01:32.11 UTC
2009-03-06 05:59:22.099 UTC 2 minutes

March 04, 2009 / 03:33:05 PM
March 04, 2009 / 03:34:48 PM 2 minutes mostly cfoutput

2009-02-16 18:18:55.518 UTC
2009-02-16 18:16:26.537 UTC 2 minutes

February 08, 2009 / 06:23:27 PM

———————————————————
WEBE22/mysqlb15 (stopped using on 1/24/09)

January 23, 2009 / 03:35:25 PM
January 23, 2009 / 03:34:25 PM 1 minute

January 23, 2009 / 03:14:25 PM 1 second

2009-01-23 09:05:59.409 UTC (around 2:30AM)
2009-01-23 09:14:33.741 UTC 8:34 minutes down

January 22, 2009 / 06:01:29 PM
January 22, 2009 / 06:00:52 PM 37 seconds (mostly cfoutput errors this time)

January 18, 2009 / 12:29:07 PM
January 18, 2009 / 12:28:16 PM 51 seconds

January 17, 2009 / 11:49:02 AM
January 17, 2009 / 11:49:04 AM 2 seconds while the query took over 30 seconds before I got the error

January 17, 2009 / 12:02:52 AM
January 17, 2009 / 12:02:17 AM 35 seconds

January 14, 2009 / 02:51:19 PM 1 second

January 14, 2009 / 12:12:33 PM
January 14, 2009 / 12:06:30 PM 6 minutes

January 14, 2009 / 12:01:29 PM 1 second

January 14, 2009 / 10:51:15 AM
January 14, 2009 / 10:51:02 AM 12 seconds

January 14, 2009 / 10:08:37 AM
January 14, 2009 / 10:07:59 AM 38 seconds

January 13, 2009 / 04:55:41 PM
January 13, 2009 / 04:55:37 PM 4 seconds

January 13, 2009 / 01:24:52 PM
January 13, 2009 / 01:24:47 PM 5 seconds

January 13, 2009 / 01:07:05 PM
January 13, 2009 / 01:06:10 PM 55 seconds

January 13, 2009 / 09:00:47 AM
January 13, 2009 / 09:00:34 AM 13 seconds

January 12, 2009 / 09:27:11 PM

January 12, 2009 / 01:00:20 PM

————————-

I was planning on sharing this at the beginning of next year, but I thought I would share now since I have had more than usual downtime in the past few days after CrystalTech’s major power outage at their datacenter.

My idea for this post is not to show that CrystalTech is bad in anyway (they are actually very good), it is mostly to compare my shared vs. dedicated outage experiences. [I will post my dedicated downtime later]

I am alerted when my site goes down via a Coldfusion script I wrote. When the entire host goes down like it did last week then my RentVine.com site switches over to another host at HostFolio via DNSMadeEasy. It changes the IP address of the site at the DNS level.

I know that living on a shared environment especially in Coldfusion 8 has its ups and downs. The normal downtime for my site on Crystaltech is a couple of minutes here and there every couple of weeks. This is acceptable to me given the amount I am spending per month.

The past couple of days I have had more than my share of downtime. Even as I write this more emails are flowing in letting me know that the site is down yet again today.

I usually only get two types of errors:

CF Error: The request has exceeded the allowable time limit Tag: cfoutput The error occurred on line 19.

CF Error: The request has exceeded the allowable time limit Tag: CFQUERY The error occurred on line 119.

These types of errors could be caused from almost anything and I am guessing they are hard to track down unless you have an admin watching a program like FusionReactor on my shared server.  I don’t think Crystaltech uses FusionReactor much because every time my site goes down and I call the tech can never tell me why the site went down. My guess is someone else on the server is hogging all the resources with a poorly written script.

A minute or two of downtime does not sound that bad, but because of my traffic on RentVine I can have hundreds of errors like I am having today.

[Note: I am not including the downtime that occurs sometimes at 2am when they mostly likely have to reboot after a patch update.]