random ios builds failing

  • 1
  • Problem
  • Updated 4 years ago
  • In Progress
I have an automated process (using the developer API) that uploads new zip files, unlocks the keys, waits for the builds to finish, then downloads the APK and IPA files. I have 6 apps. This process has NOT changed.

Recently, about 1/2 of the time one of the ios builds fail. It is not always the same app. The Android builds always work.

Note: I have "Retry" logic in my process. When I submit a build for a specific app on a specific platform, it returns correctly. But, when I query the app, it doesn't show as waiting to be built. So my process fails.

When I go to a browser and log into build.phonegap.com, all the apps show as successfully built (no failure log). I am assuming the call does not immediately queue the app/platform to be built, but it gets rebuilt before I can check the build error (on PGB).

1. Is there stability problems with the ios build server(s)?
2. Obviously I am using a paid account as I have more than one app. Is there any "Paid Support" options?
3. I have been fighting build problems of various kinds since we started using PGB. Is this service in Beta?
Photo of Shawn Winterbottom

Shawn Winterbottom

  • 29 Posts
  • 0 Reply Likes
  • extremely frustrated

Posted 5 years ago

  • 1
Photo of Amir

Amir

  • 8261 Posts
  • 263 Reply Likes
Hi Shawn

Most probably caused by capacity issue on one of our servers instance recently. We have took that our from production for repair.

Please monitor for another few days and let us know if problem persisting.

Sorry for all the troubles

Thanks
(Edited)
Photo of Shawn Winterbottom

Shawn Winterbottom

  • 29 Posts
  • 0 Reply Likes
This time EVERY app on every platform took an inordinate amount of time to build. Our automated process timed out.

1. Our automated process uploads new codes (starts the build process) then disconnects

2. Our automated process continues work on building our other products (web site, desktop program, other utility programs

3. Then our automated process connects to PGB to download the apps (will wait for a set time for the apps to finish building).

Our full automated process takes almost 30 minutes. So normally the apps are built by the time we get to the third step. I estimate it is at least 10 minutes between when we upload/start a new build and when we check to for completion to download the apps. Overall, I am guessing it took over 15 minutes for the builds to complete.

As we are PAYING for this service...this entirely unacceptable. My supervisor has already asked me to look into alternative solutions.

We need a STABLE build process!
Photo of Amir

Amir

  • 8261 Posts
  • 263 Reply Likes
Hi Shawn

How frequent did this happen? because we encounter server hiccup the moment you post the above message.

Anyway did the original issue - 'random iOS build failing' - resolved?
Photo of Shawn Winterbottom

Shawn Winterbottom

  • 29 Posts
  • 0 Reply Likes
1st build...ios failed (original issue).
2nd build...EVERYTHING failed (no way to tell if it was fixed or got worse)

We do 1-3 builds a week when we are trying to do a release. Other times we may go a full week or more without doing a build. Right now we are doing about 1 build a week. So I expect a build will be tried today or tomorrow.

FYI: When we are doing fewer builds, it is a bigger problem to have a build fail. We have to wait longer for a "Good" build, which puts our QA department behind on testing.
Photo of Yi Ming Kuan

Yi Ming Kuan

  • 1840 Posts
  • 78 Reply Likes
Hi Shawn,

Sorry to hear about that.

We have a build server status page at http://status.build.phonegap.com/. If the issue is not caused by compile errors, (e.g. stuck at pending builds, disk errors, or server errors) let us know via the email link on the status page.

-yiming
Photo of Shawn Winterbottom

Shawn Winterbottom

  • 29 Posts
  • 0 Reply Likes
I always check this page...it always shows everything is good, unless PGB is intentionally brought down.
Photo of Shawn Winterbottom

Shawn Winterbottom

  • 29 Posts
  • 0 Reply Likes
OK...Tried to do another build...and it failed AGAIN....This is 3 times in a row.

This time, all six apps built fine. But I got errors download the ipa/apk files.

The first few files downloaded just fine. But then I started getting "(504) Gateway Timeout" errors.

FYI: I did check the server status page and it shows all good.
Photo of ryan

ryan, Developer

  • 1538 Posts
  • 132 Reply Likes

3. Then our automated process connects to PGB to download the apps (will wait for a set time for the apps to finish building).

Our full automated process takes almost 30 minutes. So normally the apps are built by the time we get to the third step. I estimate it is at least 10 minutes between when we upload/start a new build and when we check to for completion to download the apps. Overall, I am guessing it took over 15 minutes for the builds to complete.


If I understand correctly (correct me if not), you're simply triggering a build, and waiting x minutes and then you're assuming the build was complete and attempting download?

If so, this really isn't how the API should be used. When you trigger a build, you need to begin polling the API to determine when the build completes. Then you should check for an error message, or a success flag. Only on success should you attempt to download the compiled app. In addition if you're automatically triggering a rebuild it makes it hard for us to help debug.

While we do utilize autoscaling, there is no way to avoid the fact that build times are going to fluctuate depending on load. You can't simply assume that the build will complete successfully in x minutes.

Additionally it would help debug here if you actually shared the code you're using to trigger a build, and download the app.
Photo of Shawn Winterbottom

Shawn Winterbottom

  • 29 Posts
  • 0 Reply Likes
I wrote a C# winform program to communicate with PGB. This program takes parameters (in the form of an XML file) that determine what steps will happen. In our automated build process, I execute this program two separate times...

Session 1:
1. Unlock appropriate keys, which will trigger a build if the key was currently locked
2. In a loop, I repeatedly check the state of the build...waiting for the build process to finish (if it started).
3. Upload new zip files, which triggers a build.
4. Program exists, leaving PGB to finish the build.

Session 2:
1. In a loop, I repeatedly check the state of the build...waiting for the build process
2. I then check for an error message/success flag for each app/platform
3. IF there is an error, I trigger a rebuild.
4. IF a rebuild was triggered, I sit in a loop checking the state of the build...waiting for the process to finish.
5. If there were no error messages, I download all the files (apk,ipa).

Some important points:
1. As you can see, by my notes, I am repeatedly checking to make sure that the builds are finished (either successfully or for an error message). I am NOT simply waiting an arbitrary amount of time, then trying to download.

2. During the loop, where I check for the builds to complete, I have a timeout built in. This way, my program will not "hang forever" if your build server dies (never finishes building). Initially, this timeout was set to 5 minutes. I just changed it to 15 minutes. This is specified as a parameter and does not require a change to my program.

3. There is about 10 minutes between when session 1 ends and session 2 starts. This means, that overall I was giving your process about 15 minutes to build. Now it is about 25 minutes. Under normal circumstances, build are finished in less than 1 minute.

4. My program, that interfaces with PGB had not changed for a long time. With the recent failures on PGB, I did make a change so if a app/platform fails, I rebuild all platforms for that app. I was just triggering a rebuild for that specific app/platform. PGB has a bug trying to trigger a rebuild for a specific app/platform. I have submitted this issue in the past. Basically, if my code doesn't change...But, sometimes PGB fails and other times it doesn't...It is very likely a problem with PGB. NOT with my app.

5. My program that interfaces with PGB is "Company Property". As such, I cannot give you the source code without signed Non-Disclosure/Non-Compete documents. That process is probably not feasible.

6. My program does some basic logging. The most recent failure was the Gateway error, I received when trying to download. Note, that this log is only from the 2nd session:

[Processing Build File] - 23/06/2015 17:09:01.387
================================================
Processor Version: 1.1.0.0
Auto Close (-c): True
Skip Errors (-s): True
Don't Auto Close If Errors Occur (-x): False
File: C:\Develop\MMP\MPMobile\Build\MobileBuild\Download.xml
File Version: 1.0

[Logging In] - 23/06/2015 17:09:01.402
================================================
Logging In...DJtxQzzcDykUgxtSzFWJ
Logged In As...mpdev@managerplus.com

[Loading Apps] - 23/06/2015 17:09:02.035
================================================
ManagerPlus Mobile (1281700)
ManagerPlus Mobile (1302879)
M+ Logs (1280433)
M+ Logs (1302870)
M+ Work Requests (839455)
M+ Work Requests (1302878)
6 Apps Loaded

[Verifying Builds] - 23/06/2015 17:09:05.457
================================================
ManagerPlus Mobile (1281700): android...Success
ManagerPlus Mobile (1281700): ios...Success
ManagerPlus Mobile (1302879): ios...Success
M+ Logs (1280433): android...Success
M+ Logs (1280433): ios...Success
M+ Logs (1302870): ios...Success
M+ Work Requests (839455): android...Success
M+ Work Requests (839455): ios...Success
M+ Work Requests (1302878): ios...Success

[Downloading Files] - 23/06/2015 17:09:07.531
================================================
Downloading file...C:\Develop\MMP\MPMobile\Build\M+Mobile.apk
Downloading file...C:\Develop\MMP\MPMobile\Build\M+Mobile.ipa
Downloading file...C:\Develop\MMP\MPMobile\Build\M+MobileQA.ipa
The remote server returned an error: (504) Gateway Timeout.
Downloading file...C:\Develop\MMP\MPMobile\Build\M+Logs.apk
The remote server returned an error: (504) Gateway Timeout.
Downloading file...C:\Develop\MMP\MPMobile\Build\M+Logs.ipa
The remote server returned an error: (504) Gateway Timeout.
Downloading file...C:\Develop\MMP\MPMobile\Build\M+LogsQA.ipa
The remote server returned an error: (504) Gateway Timeout.
Downloading file...C:\Develop\MMP\MPMobile\Build\M+WorkRequests.apk
The remote server returned an error: (504) Gateway Timeout.
Downloading file...C:\Develop\MMP\MPMobile\Build\M+WorkRequests.ipa
The remote server returned an error: (504) Gateway Timeout.
Downloading file...C:\Develop\MMP\MPMobile\Build\M+WorkRequestsQA.ipa
The remote server returned an error: (504) Gateway Timeout.
Not all files were downloaded.

Processing Complete - 23/06/2015 17:09:16.622
Photo of Shawn Winterbottom

Shawn Winterbottom

  • 29 Posts
  • 0 Reply Likes
Another log:

[Processing Build File] - 19/06/2015 08:07:51.990
================================================
Processor Version: 1.1.0.0
Auto Close (-c): True
Skip Errors (-s): True
Don't Auto Close If Errors Occur (-x): False
File: C:\Develop\MMP\MPMobile\Build\MobileBuild\Download.xml
File Version: 1.0

[Logging In] - 19/06/2015 08:07:52.005
================================================
Logging In...xxxxxxxxxxxxxxxxxx
Logged In As...yyyyyyyyyyyyyyyyy

[Loading Apps] - 19/06/2015 08:07:52.621
================================================
ManagerPlus Mobile (1281700)
ManagerPlus Mobile (1302879)
M+ Logs (1280433)
M+ Logs (1302870)
M+ Work Requests (839455)
M+ Work Requests (1302878)
6 Apps Loaded

[Waiting For Pending Builds To Finish] - 19/06/2015 08:07:54.728
================================================
Checking Build Status...
Pending Builds: 11
Timed out waiting for build(s) to finish.

Processing Complete - 19/06/2015 08:12:55.730
Photo of Shawn Winterbottom

Shawn Winterbottom

  • 29 Posts
  • 0 Reply Likes
I just did another build...and it worked. I will continue to monitor. Hopefully, things are more stabile.
Photo of Shawn Winterbottom

Shawn Winterbottom

  • 29 Posts
  • 0 Reply Likes
Failed again:
Got a "The remote server returned an error: (500) Internal Server Error." trying to unlock a key.

This key is not a new key and it has not expired. The code to unlock has not changed either. The other 8 keys (ios and android) all were unlocked just fine.
Photo of Shawn Winterbottom

Shawn Winterbottom

  • 29 Posts
  • 0 Reply Likes
I am still getting frequent errors, but a retry usually fixes it. This time the retry didn't...So I have logs to share. I am NOT including all the logs from all the failures. Hopefully, you can check that using the App Id's from my own upload utility.

This log is from my upload utility (using the API):

[Processing Build File] - 30/03/2016 15:29:03.767
================================================
Processor Version: 1.1.0.0
Auto Close (-c): True
Skip Errors (-s): True
Don't Auto Close If Errors Occur (-x): False
File: C:\Develop\MMP\MPMobile\Build\MobileBuild\Download.xml
File Version: 1.0

[Logging In] - 30/03/2016 15:29:03.783
================================================
Logging In...{hidden}
Logged In As...{hidden}

[Loading Apps] - 30/03/2016 15:29:04.328
================================================
ManagerPlus Mobile (1281700)
ManagerPlus Mobile (1302879)
M+ Logs (1280433)
M+ Logs (1302870)
M+ Work Requests (839455)
M+ Work Requests (1302878)
6 Apps Loaded

[Verifying Builds] - 30/03/2016 15:29:08.231
================================================
ManagerPlus Mobile (1281700): android...Failed
ManagerPlus Mobile (1281700): ios...Success
ManagerPlus Mobile (1302879): ios...Success
M+ Logs (1280433): android...Failed
M+ Logs (1280433): ios...Success
M+ Logs (1302870): ios...Failed
M+ Work Requests (839455): android...Failed
M+ Work Requests (839455): ios...Failed
M+ Work Requests (1302878): ios...Success

[Rebuilding Apps] - 30/03/2016 15:29:09.139
================================================
M+ Logs (1280433): android
The remote server returned an error: (422) Unprocessable Entity.

Processing Complete - 30/03/2016 15:29:10.316

For the Log from PGB, I am only including logs from App Id 839455:
Android (log only has ONE LINE):
Build Date: 2016-03-30 21:22:46 +0000


iOS:
Build Date: 2016-03-30 14:23:02 -0700
--------------------------------------------------------------------------------
PLUGIN OUTPUT
--------------------------------------------------------------------------------
Installing "com.phonegap.plugins.barcodescanner" at "2.2.0" for ios
Notice: org.apache.cordova.camera has been automatically converted to cordova-plugin-camera and fetched from npm. This is due to our old plugins registry shutting down.
Fetching plugin "cordova-plugin-camera" via npm
Installing "cordova-plugin-camera" at "2.1.1" for ios
Notice: org.apache.cordova.media has been automatically converted to cordova-plugin-media and fetched from npm. This is due to our old plugins registry shutting down.
Fetching plugin "cordova-plugin-media" via npm
Installing "cordova-plugin-media" at "2.2.0" for ios
Fetching plugin "cordova-plugin-file" via npm
Installing "cordova-plugin-file" at "4.1.1" for ios

The Android Persistent storage location now defaults to "Internal". Please check this plugins README to see if you application needs any changes in its config.xml.

If this is a new application no changes are required.

If this is an update to an existing application that did not specify an "AndroidPersistentFileLocation" you may need to add:

""

to config.xml in order for the application to find previously stored files.
Notice: org.apache.cordova.media-capture has been automatically converted to cordova-plugin-media-capture and fetched from npm. This is due to our old plugins registry shutting down.
Fetching plugin "cordova-plugin-media-capture" via npm
Installing "cordova-plugin-media-capture" at "1.2.0" for ios
Dependent plugin "cordova-plugin-file" already installed on ios.
Notice: org.apache.cordova.device has been automatically converted to cordova-plugin-device and fetched from npm. This is due to our old plugins registry shutting down.
Fetching plugin "cordova-plugin-device" via npm
Installing "cordova-plugin-device" at "1.1.1" for ios
Notice: org.apache.cordova.file has been automatically converted to cordova-plugin-file and fetched from npm. This is due to our old plugins registry shutting down.
Plugin "cordova-plugin-file" already installed on ios. Making it top-level.
Notice: org.apache.cordova.file-transfer has been automatically converted to cordova-plugin-file-transfer and fetched from npm. This is due to our old plugins registry shutting down.
Fetching plugin "cordova-plugin-file-transfer" via npm
Installing "cordova-plugin-file-transfer" at "1.5.0" for ios
Dependent plugin "cordova-plugin-file" already installed on ios.
Notice: org.apache.cordova.geolocation has been automatically converted to cordova-plugin-geolocation and fetched from npm. This is due to our old plugins registry shutting down.
Fetching plugin "cordova-plugin-geolocation" via npm
Installing "cordova-plugin-geolocation" at "2.1.0" for ios
Notice: org.apache.cordova.network-information has been automatically converted to cordova-plugin-network-information and fetched from npm. This is due to our old plugins registry shutting down.
Fetching plugin "cordova-plugin-network-information" via npm
Installing "cordova-plugin-network-information" at "1.2.0" for ios
Notice: org.apache.cordova.inappbrowser has been automatically converted to cordova-plugin-inappbrowser and fetched from npm. This is due to our old plugins registry shutting down.
Fetching plugin "cordova-plugin-inappbrowser" via npm
Installing "cordova-plugin-inappbrowser" at "1.3.0" for ios
Fetching plugin "cordova-plugin-whitelist" via npm
Installing "cordova-plugin-whitelist" at "1.2.1" for ios
--------------------------------------------------------------------------------
COMPILE OUTPUT
--------------------------------------------------------------------------------
Build settings from command line:
ARCHS = armv7 arm64
CONFIGURATION_BUILD_DIR = /project/build
CORDOVALIB = /project/CordovaLib
ENABLE_BITCODE = false
ONLY_ACTIVE_ARCH = NO
OTHER_CODE_SIGN_FLAGS = --keychain /secure.keychain
PROVISIONING_PROFILE = 9f6576f6-c78d-4403-808a-0e11a7f6a54a

=== BUILD TARGET CordovaLib OF PROJECT CordovaLib WITH THE DEFAULT CONFIGURATION (Release) ===

Check dependencies
2016-03-30 14:22:57.460 xcodebuild[89515:52536312] DVTAssertions: ASSERTION FAILURE in /Library/Caches/com.apple.xbs/Sources/IDEXcode3ProjectSupport/IDEXcode3ProjectSupport-9061/Xcode3Core/LegacyProjects/Frameworks/DevToolsCore/DevToolsCore/Foundation/Miscellaneous/XCDerivedFileCache.m:129
Details: Assertion failed: curCacheEntry == ((void*)0)
Object:
Method: -_pruneCacheToNumberOfBytes:minimumCacheEntryAgeForRemoval:
Thread: {number = 7, name = (null)}
Hints: None
Backtrace:
0 0x000000010e103c44 -[DVTAssertionHandler handleFailureInMethod:object:fileName:lineNumber:assertionSignature:messageFormat:arguments:] (in DVTFoundation)
1 0x000000010e103778 _DVTAssertionHandler (in DVTFoundation)
2 0x000000010e10398f _DVTAssertionFailureHandler (in DVTFoundation)
3 0x000000010e1038f1 _DVTAssertionFailureHandler (in DVTFoundation)
4 0x00000001115e73eb -[XCDerivedFileCache _pruneCacheToNumberOfBytes:minimumCacheEntryAgeForRemoval:] (in DevToolsCore)
5 0x00000001115e7ed7 -[XCDerivedFileCache initWithBasePath:disableCachePruning:] (in DevToolsCore)
6 0x00000001115e83e8 +[XCDerivedFileCache sharedDerivedFileCacheWithBasePath:disableCachePruning:] (in DevToolsCore)
7 0x00000001115ce691 -[PBXCompilerSpecificationGcc2_95_2 compileSourceCodeFileAtPath:ofType:toOutputDirectory:withMacroExpansionScope:] (in DevToolsCore)
8 0x00000001116f4a91 -[XCCompilerSpecificationClang compileSourceCodeFileAtPath:ofType:toOutputDirectory:withMacroExpansionScope:] (in DevToolsCore)
9 0x00000001115c90fc -[PBXCompilerSpecificationGcc computeDependenciesForInputNodes:ofType:variant:architecture:outputDirectory:withMacroExpansionScope:] (in DevToolsCore)
10 0x000000011160f4c8 -[XCCompilerSpecification computeDependenciesForFilePath:ofType:outputDirectory:withMacroExpansionScope:] (in DevToolsCore)
11 0x00000001116e2821 -[XCBuildRuleDGSnapshot(DependencyGraphCreation) computeDependenciesForFilePath:ofType:forBuildFileReference:withOutputDirectory:parameterMacros:withMacroExpansionScope:] (in DevToolsCore)
12 0x00000001116e37be -[XCBuildRuleDGSnapshot(DependencyGraphCreation) computeDependenciesForBuildFileReference:withOutputDirectory:parameterMacros:withMacroExpansionScope:] (in DevToolsCore)
13 0x00000001116d567c -[XCSourcesBuildPhaseDGSnapshot(DependencyGraphCreation) _computeDependenciesForBuildFileReference:usingBuildRule:withMacroExpansionScope:processedPaths:] (in DevToolsCore)
14 0x00000001116d7e8d -[XCSourcesBuildPhaseDGSnapshot(DependencyGraphCreation) computeDependenciesForBuildFileReference:usingBuildRule:withMacroExpansionScope:] (in DevToolsCore)
15 0x00000001116d833a -[XCSourcesBuildPhaseDGSnapshot(DependencyGraphCreation) computeDependenciesForBuildFileReference:withMacroExpansionScope:] (in DevToolsCore)
16 0x00000001116c38fc -[XCBuildPhaseDGSnapshot(DependencyGraphCreation) computeDependenciesForAllBuildFileReferencesWithMacroExpansionScope:] (in DevToolsCore)
17 0x00000001116daf1e -[XCSourcesBuildPhaseDGSnapshot(DependencyGraphCreation) computeDependenciesForAllBuildFileReferencesWithMacroExpansionScope:] (in DevToolsCore)
18 0x00000001116dc2fc -[XCSourcesBuildPhaseDGSnapshot(DependencyGraphCreation) computeDependenciesWithMacroExpansionScope:] (in DevToolsCore)
19 0x0000000111611f81 -[XCProductTypeSpecification computeDependenciesWithMacroExpansionScope:] (in DevToolsCore)
20 0x00000001116bc227 -[XCNativeTargetDGSnapshot(DependencyGraphCreation) computeDependenciesWithMacroExpansionScope:] (in DevToolsCore)
21 0x000000011158e77b -[PBXTargetBuildContext createDependencyGraphWithTargetDGSnapshot:] (in DevToolsCore)
22 0x00000001115aa99d -[PBXTargetBuildContext(DependencyGraphEvents) dg_setTargetSnapshot:] (in DevToolsCore)
23 0x00000001115aa062 -[PBXTargetBuildContext(DependencyGraphEvents) processDependencyGraphEvents] (in DevToolsCore)
24 0x0000000111609c0e -[XCBlockQueue _processNextBlockInThreadSlotNumber:] (in DevToolsCore)
25 0x0000000111609e90 -[XCBlockQueue _processBlocksInThreadSlotNumber:] (in DevToolsCore)
26 0x00007fff882e9e92 __NSThread__main__ (in Foundation)
27 0x00007fff88cb805a _pthread_body (in libsystem_pthread.dylib)
28 0x00007fff88cb7fd7 _pthread_body (in libsystem_pthread.dylib)
29 0x00007fff88cb53ed thread_start (in libsystem_pthread.dylib)