Help get this topic noticed by sharing it on Twitter, Facebook, or email.
I’m super sad.

An error has occurred while fetching rows from salesforce.com: Failed to read from reply.

I just started getting this error message today and our SalesForce admin says that there weren't any changes. This latest try took 8 hours before it failed.

Any ideas? All our other tables are still refreshing.

Thanks!

--- Starting SF_Refresh for lead
13:38:14: Using Schema Error Action of yes
13:38:15: Using last run time of 2013-06-20 14:35:00
OLE DB provider "DBAmp.DBAmp" for linked server "salesforce" returned message "Error 1 : An error has occurred while fetching rows from salesforce.com: Failed to read from reply.
Win32Error::
The connection with the server was terminated abnormally
".
Msg 7346, Level 16, State 2, Line 1
Cannot get the data of the row from the OLE DB provider "DBAmp.DBAmp" for linked server "salesforce".
1 person has
this question
+1
Reply
  • "The connection with the server was terminated abnormally"

    This message is from your firewall/proxy terminating the connection to salesforce due to abandonment after 8 hours.

    From the message output, it looks like you have not run a sf_refresh of Lead since June 20. This is too long to wait between runs because the salesforce server only retains deleted row information for 15 days.

    Therefore you must run sf_refresh at least once every 14 days and preferably more often than that.

    Your only choice now is to run a sf_replicate (not sf_refresh) of the Lead table.

    I would run this at night when the salesforce server load is light.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. sad, anxious, confused, frustrated indifferent, undecided, unconcerned kidding, amused, unsure, silly happy, confident, thankful, excited

  • I’m relieved
    That's the strange thing- we run the refresh every two hours and it was working fine until yesterday. We had a lot of tables in our salesforce backup database that looked like this, "[dbo].[Lead_Delta2013-06-19T11:03:28.603]", so I deleted them all in addition to Lead and Lead_Previous.

    This morning I got confirmation that our offshore SFDC admin _did_ add some new formula columns. After hiding those from our DBAmp user, I was finally able to get a Replicate job through and anticipate that the refreshes will continue smoothly.

    Thanks for the quick response!
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. sad, anxious, confused, frustrated kidding, amused, unsure, silly indifferent, undecided, unconcerned happy, confident, thankful, excited

  • I’m sad again
    Actually - looks like we're still having the issue- this is the result from SF_Replicate:

    --- Starting SF_Replicate for lead
    13:57:09: Drop lead_Previous if it exists.
    13:57:09: Create lead_Previous with new structure.
    13:57:12: Run the DBAmp.exe program.
    13:57:12: DBAmp Bulk Operations. V2.15.2 (c) Copyright 2006-2012 forceAmp.com LLC
    13:57:12: Populating local table lead_Previous , AUS-DWAPP-01 / salesforce backups .
    13:57:13: Opening SQL Server rowset
    15:17:52: Error: An error has occurred while fetching rows from salesforce.com.
    15:17:52: Failed to read from reply.
    Win32Error::
    The connection with the server was terminated abnormally

    15:17:52: Error: Replication failed.
    Source: ForceAmpAPI.SQueryResults4.1
    Description: Failed to read from reply.
    Win32Error::
    The connection with the server was terminated abnormally

    15:20:20: Error: DBAmp.exe was unsuccessful.
    15:20:20: Error: Command string is C:\"Program Files"\DBAmp\DBAmp.exe Export "lead_Previous" "AUS-DWAPP-01" "salesforce backups" "salesforce"
    --- Ending SF_Replicate. Operation FAILED.
    Msg 50000, Level 16, State 1, Procedure SF_Replicate, Line 214
    --- Ending SF_Replicate. Operation FAILED.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. sad, anxious, confused, frustrated kidding, amused, unsure, silly indifferent, undecided, unconcerned happy, confident, thankful, excited

  • DBAmp makes a request of salesforce and sometimes must wait over 30 minutes for the reply from the salesforce server. Normally this error occurs when your firewall / proxy watches the request go out to the server and then determines that it can no longer wait for a response from salesforce because it considers the connection abandoned.

    You will need to work with your network group and ask them to take a look at the abandonment timeout on the firewall/proxy and adjust as needed. They may need to place a trace on the activity to determine the request that is causing the long response.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. sad, anxious, confused, frustrated indifferent, undecided, unconcerned kidding, amused, unsure, silly happy, confident, thankful, excited

  • I’m stressed
    I got our network group involved and they're not seeing any errors/timeouts. Nothing in the SQL logs, either. They've set up some monitors to see if there's issues during the timeouts.

    I've updated to the latest version and tried bulkapi- that failed, too, saying that the timeout had been reached. I'm trying it again on another table to see if that one goes through.

    It's seemingly random for our larger tables now- earlier today Contact failed with SF_Replicate, then it went through later. Some time out right away and others wait hours to time out.

    Our network folks and I are stumped.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. sad, anxious, confused, frustrated kidding, amused, unsure, silly indifferent, undecided, unconcerned happy, confident, thankful, excited

  • Can I see the sf_bulkops output ?

    You won't see anything in the SQL logs; it is all happening in the network layer.

    What firewall / proxy are you using ?
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. sad, anxious, confused, frustrated kidding, amused, unsure, silly indifferent, undecided, unconcerned happy, confident, thankful, excited

  • There are about 5 different timeouts that can occur. Can you post some actual error messages from the other larger tables ?
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. sad, anxious, confused, frustrated indifferent, undecided, unconcerned kidding, amused, unsure, silly happy, confident, thankful, excited

  • Running this: exec SF_Replicate 'salesforce', 'opportunity', bulkapi
    Gave me this:
    --- Starting SF_Replicate for opportunity
    17:56:38: Drop opportunity_Previous if it exists.
    17:56:38: Create opportunity_Previous with new structure.
    17:56:39: Run the DBAmp.exe program.
    17:56:40: DBAmp Bulk Operations. V2.17.5 (c) Copyright 2006-2013 forceAmp.com LLC
    17:56:40: Populating local table opportunity_Previous , AUS-DWAPP-01 / salesforce backups .
    17:56:40: DBAmp is using the SQL Native Client.
    17:56:40: Opening SQL Server rowset
    17:56:41: Using the bulkapi with polling every 60 seconds for up to 1200 seconds.
    17:56:42: Job still running.
    17:56:57: Job still running.
    17:57:57: Job still running.
    17:58:57: Job still running.
    17:59:58: Job still running.
    18:00:58: Job still running.
    18:01:58: Job still running.
    18:02:58: Job still running.
    18:03:59: Job still running.
    18:04:59: Job still running.
    18:05:59: Job still running.
    18:07:00: Job still running.
    18:08:03: Job still running.
    18:09:03: Job still running.
    18:10:03: Job still running.
    18:11:04: Job still running.
    18:12:04: Job still running.
    18:13:04: Job still running.
    18:14:04: Job still running.
    18:15:05: Job still running.
    18:16:05: Job still running.
    18:17:05: Job still running.
    18:17:05: Error: Polling timeout reached. Unable to retrieve results from salesforce.
    18:17:05: Error: DBAmp.exe was unsuccessful.
    18:17:05: Error: Command string is C:\"Program Files"\DBAmp\DBAmp.exe Export:bulkapi "opportunity_Previous" "AUS-DWAPP-01" "salesforce backups" "salesforce"
    --- Ending SF_Replicate. Operation FAILED.
    Msg 50000, Level 16, State 1, Procedure SF_Replicate, Line 244
    --- Ending SF_Replicate. Operation FAILED.

    -----------------------------------------------------------------------------------------

    Running this:exec SF_Replicate 'salesforce', 'Opportunity'
    Gave me this:--- Starting SF_Replicate for Opportunity

    17:14:21: Drop Opportunity_Previous if it exists.
    17:14:21: Create Opportunity_Previous with new structure.
    17:14:21: Run the DBAmp.exe program.
    17:14:21: DBAmp Bulk Operations. V2.17.5 (c) Copyright 2006-2013 forceAmp.com LLC
    17:14:21: Populating local table Opportunity_Previous , AUS-DWAPP-01 / salesforce backups .
    17:14:22: DBAmp is using the SQL Native Client.
    17:14:22: Opening SQL Server rowset
    17:54:24: Error: An error has occurred while fetching rows from salesforce.com.
    17:54:24: Failed to read from reply.
    Win32Error::
    The connection with the server was terminated abnormally

    17:54:24: Error: Replication failed.
    Source: ForceAmpAPI.SQueryResults4.1
    Description: Failed to read from reply.
    Win32Error::
    The connection with the server was terminated abnormally

    17:55:15: Error: DBAmp.exe was unsuccessful.
    17:55:15: Error: Command string is C:\"Program Files"\DBAmp\DBAmp.exe Export "Opportunity_Previous" "AUS-DWAPP-01" "salesforce backups" "salesforce"
    --- Ending SF_Replicate. Operation FAILED.
    Msg 50000, Level 16, State 1, Procedure SF_Replicate, Line 244
    --- Ending SF_Replicate. Operation FAILED.

    -----------------------------------------------------------------------------------------
    I don't know about the firewall- I'll ask and post in another comment.

    Thanks. :)
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. sad, anxious, confused, frustrated indifferent, undecided, unconcerned kidding, amused, unsure, silly happy, confident, thankful, excited

  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. happy, confident, thankful, excited indifferent, undecided, unconcerned kidding, amused, unsure, silly sad, anxious, confused, frustrated

  • Do not use the bulkapi option for sf_replicate. For large tables such as yours the job on salesforce will not complete.

    Back to the initial issue, based on the output of the sf_replicate of the Opportunity table, I would say that your firewall connection timeout is 40 mins (17:54 - 17;14). How many rows are there in your opportunity table ( look on salesforce in the Storage section) ?
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. sad, anxious, confused, frustrated kidding, amused, unsure, silly indifferent, undecided, unconcerned happy, confident, thankful, excited

  • There's 260,038 rows in the Opportunity table, so it's not terribly big.

    Last night, the SF_Replicate on our Lead table (3,766,809 rows) successfully completed running from 22:08 to 02:45 in the morning.

    IT has looked at logs during the failure time sets I've sent them and they're not seeing anything out of the ordinary. I thought you were on to something about 40 minutes, but the failure times are varying - here's the recent runs for SF_Replicate on Opportunity:
    08:06:14 - 08:31:28
    09:31:46 - 10:00:55
    10:19:33 - 12:21:52
    12:24:15 - 12:48:41
    13:11:27 - 13:19:49

    Here's the latest failure results:
    --- Starting SF_Replicate for Opportunity
    13:11:27: Drop Opportunity_Previous if it exists.
    13:11:27: Create Opportunity_Previous with new structure.
    13:11:27: Run the DBAmp.exe program.
    13:11:27: DBAmp Bulk Operations. V2.17.5 (c) Copyright 2006-2013 forceAmp.com LLC
    13:11:27: Populating local table Opportunity_Previous , AUS-DWAPP-01 / salesforce backups .
    13:11:28: DBAmp is using the SQL Native Client.
    13:11:28: Opening SQL Server rowset
    13:19:18: Error: An error has occurred while fetching rows from salesforce.com.
    13:19:18: Failed to read from reply.
    Win32Error::
    The connection with the server was terminated abnormally

    13:19:18: Error: Replication failed.
    Source: ForceAmpAPI.SQueryResults4.1
    Description: Failed to read from reply.
    Win32Error::
    The connection with the server was terminated abnormally

    13:19:49: Error: DBAmp.exe was unsuccessful.
    13:19:49: Error: Command string is C:\"Program Files"\DBAmp\DBAmp.exe Export "Opportunity_Previous" "AUS-DWAPP-01" "salesforce backups" "salesforce"
    --- Ending SF_Replicate. Operation FAILED.
    Msg 50000, Level 16, State 1, Procedure SF_Replicate, Line 244
    --- Ending SF_Replicate. Operation FAILED.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. sad, anxious, confused, frustrated indifferent, undecided, unconcerned kidding, amused, unsure, silly happy, confident, thankful, excited

  • Can you check with the network group and see if there has been any recent changes to the SSL Certificates ? For example, were the 128 bit ones replaced with 256 bit cert. recently ?

    We continue to believe that there is something happening in the network layer that is causing the issue. Has the trace was put in place at the firewall for the traffic headed to salesforce yielded anything ?
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. sad, anxious, confused, frustrated kidding, amused, unsure, silly indifferent, undecided, unconcerned happy, confident, thankful, excited

  • I’m exhausted, but happy.
    Turns out we had some hardware issues on our end. Everything seems to be working now and I'm reluctant to try to get it to fail again.

    Thanks so much for your help, Bill, and sorry for the fire drill.

    Anytime I get the appropriate audience and opportunity I tell people how awesome a product DBAmp is and what amazing support you give.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. sad, anxious, confused, frustrated kidding, amused, unsure, silly indifferent, undecided, unconcerned happy, confident, thankful, excited

  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. sad, anxious, confused, frustrated kidding, amused, unsure, silly indifferent, undecided, unconcerned happy, confident, thankful, excited